Of course, if a hypervisor is already available for the board, then it
would be just as easy to use a paravirtualized interface, e.g. Xen's
HYPERVISOR_console_io hypercall. But somebody has to port the
hypervisor first :-)

Of course a console is needed. But I was questioning the need for something much
more elaborate than a putchar/getchar interface. I understand its needed to port
the hypervisor but I undersatnd it would be needed for very early parts of the hypervisor where it does not even have the ability to master a uart on its own.

regards
ved

On Thu, 2 Jun 2022, Vedvyas Shanbhogue via lists.riscv.org wrote:

Based on discussion it did not seem like it needs to be much fancier than
this as this is for early OS/VMM code till it has enough functionality to
directly interact with a uart.

The goal of the shared memory based SBI call for early prints is to
minimize the number of traps which in-turn helps virtualization to
drastically reduce boot-time.

Understand that better now. But if that is the main motivation then
I am not understanding why we would want to push all of this into
M-mode firmware vs. defining a set of standardized pv-ops to be used
by guest OSes.

That is because it is useful to have debug console output when porting a
hypervisor or baremetal code to a new board.

Of course, if a hypervisor is already available for the board, then it
would be just as easy to use a paravirtualized interface, e.g. Xen's
HYPERVISOR_console_io hypercall. But somebody has to port the
hypervisor first :-)

On Thu, 2 Jun 2022, Schwarz, Konrad via lists.riscv.org wrote:

Hi Anup,

From: sig-hypervisors@... <sig-hypervisors@...> On Behalf Of Anup Patel via
lists.riscv.org
Subject: [sig-hypervisors] SBI Debug Console Extension Proposal (Draft v1)

Below is the draft proposal for SBI Debug Console Extension.

Here are my thoughts:

* Guest memory access: I think this would be the first SBI extension to require access to
guest memory. This needs to be considered carefully, but I think the higher bandwidth afforded by
the interface is useful enough to allow this.

* API:
* Currently, only a write interface is provided. It would be much better to have a
read/write interface.

Benefits of this would be to allow a hypervisor to control an OS, e.g., for testing purposes
or to automate installation tasks. Inter-guest communication could also be realized
via such an interface.

This could be done. As an example Xen provides the hypercall
HYPERVISOR_console_io. The first parameter is the operation:
CONSOLEIO_write or CONSOLEIO_read.

The interface in RISC-V spec language would be something along these
lines:

struct sbiret sbi_debug_console_io(unsigned long operation, /* read or write */
unsigned long address,
unsigned long num_chars)

There is no memory area pre-registration required, however appropriate
checks on the validity of the address provided should always be done in
the implementation.

The good thing about getting rid of the pre-registration is that
multiple threads could make concurrent sbi_debug_console_io requests on
different CPUs.

I think it might be a good idea in the guest-side implementation (e.g.
Linux or Zephyr) to choose a specific memory area for this. However, it
doesn't have to be part of the interface. I think it should be an
implementation detail.

The firmware/hypervisor-side implementation of sbi_debug_console_io
can deal with any addresses provided as long as they are valid.

On Thu, Jun 2, 2022 at 8:18 AM Anup Patel <apatel@...> wrote:

On Thu, Jun 2, 2022 at 8:30 PM Ved Shanbhogue <ved@...> wrote:

On Thu, Jun 02, 2022 at 07:52:55PM +0530, Anup Patel wrote:

On Thu, Jun 2, 2022 at 6:29 PM Ved Shanbhogue <ved@...> wrote:
This is very slow for virtualized world particularly KVM RISC-V because each
SBI v0.1 putchar() or getchar() will trap to KVM RISC-V and KVM RISC-V will
forward it to user-space QEMU or KVMTOOL. This means each early print
character using SBI v0.1 putchar() will go all the way to host user-space and
come back. This is horribly slow for KVM Guest. This becomes further slower
for nested virtualization.

Since this is for debug and really early phase debug till enough of the guest
boots up to use a VFIO based char driver provided by the VMM, I am not sure
that the slowness matters.
Even this SBI call I expected the VMM to intercept and if the VMM has all
emulation in the user space VMM - e.g. the console tty is open by the user
space VMM, I am not sure this would avoid that trip to user space.
If the motivation is primarily VM debug then perhaps a standardized set of
hypercalls implemented by KVM makes more sense than SBI calls that would
need to be built into the M-mode firmware?

We have a requirement in the OS-A platform spec to mandate a particular type
of UART for early prints but there were objections on selecting one type of UART
over another.

This SBI debug console extension is also a way to avoid standardizing a
particular type of UART in OS-A platform spec.

I worry about bugs/security issues that can be caused by M-mode firmware accessing strings in untrusted memory.

The VirtIO based para-virt devices rely heavily on shared memory so I think
it is possible to address security concerns related to shared memory.

Yes, and now that I understand the motivation better why dont we define this
as a hypercall/pv-ops interface to a VMM than a SBI call to the M-mode
firmware and needing to build a virt-io like framework in firmware.

The VirtIO framework has rings in shared memory but over here it is just
shared memory without any formatted data structure.

The API as defined does not say whether the address is a virtual address or a physical address.

It is a physical address. I will clarify this in Draft v2.

Thanks. I was not sure since we had all the discussion about Svpbmt.

Based on discussion it did not seem like it needs to be much fancier than this as this is for early OS/VMM code till it has enough functionality to directly interact with a uart.

The goal of the shared memory based SBI call for early prints is to
minimize the number of traps which in-turn helps virtualization to
drastically reduce boot-time.

Understand that better now. But if that is the main motivation then
I am not understanding why we would want to push all of this into
M-mode firmware vs. defining a set of standardized pv-ops to be used
by guest OSes.

The SBI debug console also provides a standard way of early prints for
supervisor software running natively (directly under M-mode) irrespective
of the type of UART/Console available on the console so it's not just
for a virtualized world. Although, the virtualized world has an added
performance advantage due to reduced traps.

It's also helpful in early board bringup and early debugging as
pointed out by Heiko as well.
To address some of the concerns raised above, how about putting some
restrictions on sbi_debug_console_set_area ?

1. Instead of any size, restrict the shared memory region to a fixed
size (256 byte should be enough for early prints) ?
It is still a common area across all harts. Thus, atomicity needs to
be ensured by the M-mode or HS mode.
We may explore setting up a per-heart area to avoid synchronization
issues but platforms with large hart counts may have to
share multiple pages depending on the page size.

2. The shared memory region should be set up only once during a boot
cycle. Any further invocation of sbi_debug_console_set_area
should return appropriate errors.

If the per heart shared memory region approach is preferred, the above
restriction applies once per hart.

Regards,
Anup

On Thu, Jun 2, 2022 at 8:30 PM Ved Shanbhogue <ved@...> wrote:

On Thu, Jun 02, 2022 at 07:52:55PM +0530, Anup Patel wrote:

On Thu, Jun 2, 2022 at 6:29 PM Ved Shanbhogue <ved@...> wrote:
This is very slow for virtualized world particularly KVM RISC-V because each
SBI v0.1 putchar() or getchar() will trap to KVM RISC-V and KVM RISC-V will
forward it to user-space QEMU or KVMTOOL. This means each early print
character using SBI v0.1 putchar() will go all the way to host user-space and
come back. This is horribly slow for KVM Guest. This becomes further slower
for nested virtualization.

Since this is for debug and really early phase debug till enough of the guest
boots up to use a VFIO based char driver provided by the VMM, I am not sure
that the slowness matters.
Even this SBI call I expected the VMM to intercept and if the VMM has all
emulation in the user space VMM - e.g. the console tty is open by the user
space VMM, I am not sure this would avoid that trip to user space.
If the motivation is primarily VM debug then perhaps a standardized set of
hypercalls implemented by KVM makes more sense than SBI calls that would
need to be built into the M-mode firmware?

We have a requirement in the OS-A platform spec to mandate a particular type
of UART for early prints but there were objections on selecting one type of UART
over another.

This SBI debug console extension is also a way to avoid standardizing a
particular type of UART in OS-A platform spec.

I worry about bugs/security issues that can be caused by M-mode firmware accessing strings in untrusted memory.

The VirtIO based para-virt devices rely heavily on shared memory so I think
it is possible to address security concerns related to shared memory.

Yes, and now that I understand the motivation better why dont we define this
as a hypercall/pv-ops interface to a VMM than a SBI call to the M-mode
firmware and needing to build a virt-io like framework in firmware.

The VirtIO framework has rings in shared memory but over here it is just
shared memory without any formatted data structure.

The API as defined does not say whether the address is a virtual address or a physical address.

It is a physical address. I will clarify this in Draft v2.

Thanks. I was not sure since we had all the discussion about Svpbmt.

Based on discussion it did not seem like it needs to be much fancier than this as this is for early OS/VMM code till it has enough functionality to directly interact with a uart.

The goal of the shared memory based SBI call for early prints is to
minimize the number of traps which in-turn helps virtualization to
drastically reduce boot-time.

Understand that better now. But if that is the main motivation then
I am not understanding why we would want to push all of this into
M-mode firmware vs. defining a set of standardized pv-ops to be used
by guest OSes.

The SBI debug console also provides a standard way of early prints for
supervisor software running natively (directly under M-mode) irrespective
of the type of UART/Console available on the console so it's not just
for a virtualized world. Although, the virtualized world has an added
performance advantage due to reduced traps.

Regards,
Anup

Mark,

In case you missed it from the agenda-email: we have the formation of
a Firmware & Platforms Services SIG on the agenda for the (joint)
Software HCs meeting today. Atish has kindly volunteered to be the
acting chair and we'll announce as soon as the draft charter is done.
The scope of the new SIG will be (at least) SBI, ACPI, UEFI and
bootloaders.

Philipp.

On Thu, Jun 2, 2022 at 7:41 AM Aaron Durbin <adurbin@...> wrote:

On Thu, Jun 2, 2022 at 8:38 AM Anup Patel <apatel@...> wrote:

On Thu, Jun 2, 2022 at 7:36 PM mark <markhimelstein@...> wrote:

we have seen a number of SBI changes proposed (debug console, ap tee, ...).

will there be one big rev ala priv 1.12, small fast tracks , something else?

I also suggest that this either needs to be run out of the priv sw HC or convene a new TG. Can we conduct this conversation there (and create the appropriate group or committee mail aliases).

My understanding is that different TGs or SIGs will propose their own
SBI extensions. Once a set of changes for next SBI spec release are
finalized, the SBI TG can collect all new SBI extensions and prepare a
draft of the next SBI spec. It will be the responsibility of SBI TG to
complete the ratification process.

As per my discussions with Philip, there will be a Firmware & Platform
Services SIG
which will cover all the platform services specs (SBI, UEFI, ACPI).

It can spin off separate TGs for SBI/ACPI which will be actually
responsible to deliver
the next version of the SBI specification.

Is this understanding correct ?

That aligns w/ my recollection on how we'll handle updates to SBI.

Yup. That's what I was thinking as well. Fortunately, a number of new
extensions have started to take shape.
Instead of releasing a version of each new extension, it would be good
to have a release that combines all of them.

Regards,
Anup

Thanks
Mark

--------
sent from a mobile device. please forgive any typos.

On Thu, Jun 02, 2022 at 07:52:55PM +0530, Anup Patel wrote:

On Thu, Jun 2, 2022 at 6:29 PM Ved Shanbhogue <ved@...> wrote:
This is very slow for virtualized world particularly KVM RISC-V because each
SBI v0.1 putchar() or getchar() will trap to KVM RISC-V and KVM RISC-V will
forward it to user-space QEMU or KVMTOOL. This means each early print
character using SBI v0.1 putchar() will go all the way to host user-space and
come back. This is horribly slow for KVM Guest. This becomes further slower
for nested virtualization.

Since this is for debug and really early phase debug till enough of the guest
boots up to use a VFIO based char driver provided by the VMM, I am not sure
that the slowness matters.
Even this SBI call I expected the VMM to intercept and if the VMM has all emulation in the user space VMM - e.g. the console tty is open by the user
space VMM, I am not sure this would avoid that trip to user space. If the motivation is primarily VM debug then perhaps a standardized set of
hypercalls implemented by KVM makes more sense than SBI calls that would
need to be built into the M-mode firmware?

I worry about bugs/security issues that can be caused by M-mode firmware accessing strings in untrusted memory.

The VirtIO based para-virt devices rely heavily on shared memory so I think
it is possible to address security concerns related to shared memory.

Yes, and now that I understand the motivation better why dont we define this as a hypercall/pv-ops interface to a VMM than a SBI call to the M-mode firmware and needing to build a virt-io like framework in firmware.

The API as defined does not say whether the address is a virtual address or a physical address.

It is a physical address. I will clarify this in Draft v2.

Thanks. I was not sure since we had all the discussion about Svpbmt.

Based on discussion it did not seem like it needs to be much fancier than this as this is for early OS/VMM code till it has enough functionality to directly interact with a uart.

The goal of the shared memory based SBI call for early prints is to
minimize the number of traps which in-turn helps virtualization to
drastically reduce boot-time.

Understand that better now. But if that is the main motivation then
I am not understanding why we would want to push all of this into
M-mode firmware vs. defining a set of standardized pv-ops to be used
by guest OSes.

regards
ved

On Thu, Jun 2, 2022 at 7:36 PM mark <markhimelstein@...> wrote:

we have seen a number of SBI changes proposed (debug console, ap tee, ...).

will there be one big rev ala priv 1.12, small fast tracks , something else?

I also suggest that this either needs to be run out of the priv sw HC or convene a new TG. Can we conduct this conversation there (and create the appropriate group or committee mail aliases).

My understanding is that different TGs or SIGs will propose their own
SBI extensions. Once a set of changes for next SBI spec release are
finalized, the SBI TG can collect all new SBI extensions and prepare a
draft of the next SBI spec. It will be the responsibility of SBI TG to
complete the ratification process.

Is this understanding correct ?

Regards,
Anup

Thanks
Mark

--------
sent from a mobile device. please forgive any typos.

On Thu, Jun 2, 2022 at 6:29 PM Ved Shanbhogue <ved@...> wrote:

Should we keep this simple in the SBI - only have register based inputs - to send and receive 1 byte in each call?
Keeping it a simple out_byte or in_byte - a serial port like interface seems the simplest and most secure.

The legacy SBI v0.1 putchar() and getchar() are byte-level send/receive calls.

This is very slow for virtualized world particularly KVM RISC-V because each
SBI v0.1 putchar() or getchar() will trap to KVM RISC-V and KVM RISC-V will
forward it to user-space QEMU or KVMTOOL. This means each early print
character using SBI v0.1 putchar() will go all the way to host user-space and
come back. This is horribly slow for KVM Guest. This becomes further slower
for nested virtualization.

The MMIO based early prints are further worse because we have at least two
MMIO traps for every character where each MMIO exits to host user-space.

The shared memory based SBI puts() drastically reduces the number of SBI
traps hence reducing boot time with early prints enabled.

I worry about bugs/security issues that can be caused by M-mode firmware accessing strings in untrusted memory.

The VirtIO based para-virt devices rely heavily on shared memory so I think
it is possible to address security concerns related to shared memory.

The API as defined does not say whether the address is a virtual address or a physical address.

It is a physical address. I will clarify this in Draft v2.

If it is a virtual address then the SBI call will need to use a MPRV load/store to gather this data and will also need to deal with page fault, access faults, etc. that may occur on such accesses.

Yes, MPRV (or HLV/HSV) based load/store have performance issues
particularly due to page faults. These are prevalent in some of the
legacy SBI v0.1 calls. With SBI v0.2 (or higher), we have tried to
ensure that we don't use virtual addresses as parameter in newer SBI
calls.

Based on discussion it did not seem like it needs to be much fancier than this as this is for early OS/VMM code till it has enough functionality to directly interact with a uart.

The goal of the shared memory based SBI call for early prints is to
minimize the number of traps which in-turn helps virtualization to
drastically reduce boot-time.

Regards,
Anup

regards
ved



On Thu, Jun 2, 2022 at 7:43 AM Anup Patel <apatel@...> wrote:

On Thu, Jun 2, 2022 at 2:58 PM Heiko Stübner <heiko@...> wrote:

Am Donnerstag, 2. Juni 2022, 10:50:56 CEST schrieb Heiko Stübner:

Am Donnerstag, 2. Juni 2022, 10:47:58 CEST schrieb Anup Patel:

On Thu, Jun 2, 2022 at 12:02 AM Heiko Stübner <heiko@...> wrote:

Hi Anup,

Am Mittwoch, 1. Juni 2022, 18:17:32 CEST schrieb Anup Patel:

Hi All,

Below is the draft proposal for SBI Debug Console Extension.

Please review it and provide feedback.

Thanks,
Anup

Debug Console Extension (EID #0x4442434E "DBCN")
================================================

The debug console extension defines a generic mechanism for boot-time
early prints from supervisor-mode software which allows users to catch
boot-time issues in supervisor-mode software.

This extension replaces legacy console putchar (EID #0x01) extension
and it is better in following ways:
1) It follows the new calling convention defined for SBI v1.0
(or higher).
2) It is based on a shared memory area between SBI implementation
and supervisor-mode software so multiple characters can be
printed using a single SBI call.

The supervisor-mode software must set the shared memory area before
printing characters on the debug console. Also, all HARTs share the
same shared memory area so only one HART needs to set it at boot-time.

Function: Set Console Area (FID #0)
-----------------------------------

struct sbiret sbi_debug_console_set_area(unsigned long addr_div_by_4,
unsigned long size)

Set the shared memory area specified by `addr_div_by_2` and `size`

typo in the "div_by_2" (not 4 like below and in the function itself) ?

parameters. The `addr_div_by_4` parameter is base address of the
shared memory area right shifted by 2 whereas `size` parameter is
the size of shared memory area in bytes.

The shared memory area should be normal cacheable memory for the
supervisor-mode software. Also, the shared memory area is global
across all HARTs so SBI implementation must ensure atomicity in
setting the shared memory area.

Errors:
SBI_SUCCESS - Shared memory area set successfully.
SBI_ERR_INVALID_ADDRESS - The shared memory area pointed by
`addr_div_by_2` and `size` parameters
is not normal cacheable memory or not
accessible to supervisor-mode software.

Function: Console Puts (FID #1)
-------------------------------

struct sbiret sbi_debug_console_puts(unsigned long area_offset,
unsigned long num_chars)

Print the string specified by `area_offset` and `num_chars` on
the debug console. The `area_offset` parameter is the start of
string in the shard memory area whereas `num_chars` parameter
is the number of characters (or bytes) in the string.

This is a blocking SBI call and will only return after printing
all characters of the string.

Errors:
SBI_SUCCESS - Characters printed successfully.
SBI_ERR_INVALID_ADDRESS - The start of the string (i.e.
`area_offset`) or end of the string
(i.e. `area_offset + num_chars`) is
outside shared memory area.

This will vastly reduce the number of needed ecalls when outputting
characters, so this will probably improve performance quite a bit :-)


I guess I still would like to have an _additional_ single-character
putc call. As mentioned in the other thread [0], especially on consumer
hardware [where there is no elaborate debug infrastructure] this can
be a very handy debugging tool even in the earliest stages of a
booting kernel (both before relocation and even inside the startup
assembly).

I.e. just doing a
li a7, 1
li a6, 0
li a0, 36
ecall

in any kernel assembly will just output a "$" character right now, without
needing any preparation at all - same with using the current
sbi_console_putchar() directly in c-code.

This _can_ be very helpful in some cases, so I guess it would be nice
to keep such a functionality around also in the new spec.

You can easily create multiple pre-populated strings using ".asciiz" in
assembly sources. Just set the base address of pre-populated strings
once on boot hart and print from anywhere using usual 4-5 instruction
(similar to what posted above).

ok, sounds like a plan as well :-)

though, how does that relate to the time before MMU setup?

I.e. in response to Heinrich's mail you talk about svpbmt, so I guess you
expect virtual memory there, so what is the expected value-type before
the mmu is setup in S-mode ?

The memory type should be 0 (i.e. PMA) for the shared memory between
SBI implementation and supervisor software. Before MMU setup, the
memory type is by default 0 (i.e. PMA) so we don't have to mandate
any memory type for MMU disabled case.

We only have issue on systems with Svpmbt where supervisor software
can potentially map the shared memory as non-cacheable or IO (memory
type != 0) using PTE memory type bits.

In addition to above, a SBI implementation must ensure that the shared
memory address provided by supervisor software is a regular memory
(not MMIO device). This can be easily achieved in OpenSBI, KVM RISC-V,
and various hypervisors.

Regards,
Anup

[0] http://lists.infradead.org/pipermail/opensbi/2022-June/002796.html

Regards,
Anup

we have seen a number of SBI changes proposed (debug console, ap tee, ...).

will there be one big rev ala priv 1.12, small fast tracks , something else?

I also suggest that this either needs to be run out of the priv sw HC or convene a new TG. Can we conduct this conversation there (and create the appropriate group or committee mail aliases).

Thanks
Mark

--------
sent from a mobile device. please forgive any typos.

Hello!
Glad to see a new proposal to the SBI standard. Besides to discussion before, I may (if possible or proper) suggest an idea of not implementing this extension, for SBI should provide transparency of peripherals to kernel other than wrap every possible drivers in SBI itself. Am I totally wrong, or are there any further ideas to this topic? Thanks :)

Best Regards,
Luo Jia

On Thu, Jun 2, 2022 at 2:43 PM Heinrich Schuchardt
<heinrich.schuchardt@...> wrote:

On 6/2/22 10:44, Anup Patel wrote:

On Wed, Jun 1, 2022 at 11:59 PM Heinrich Schuchardt
<heinrich.schuchardt@...> wrote:

On 6/1/22 18:17, Anup Patel wrote:

Hi All,

Below is the draft proposal for SBI Debug Console Extension.

Please review it and provide feedback.

Thanks,
Anup

Debug Console Extension (EID #0x4442434E "DBCN")
================================================

The debug console extension defines a generic mechanism for boot-time
early prints from supervisor-mode software which allows users to catch
boot-time issues in supervisor-mode software.

This extension replaces legacy console putchar (EID #0x01) extension
and it is better in following ways:

Thanks for starting to close this gap.

1) It follows the new calling convention defined for SBI v1.0
(or higher).
2) It is based on a shared memory area between SBI implementation
and supervisor-mode software so multiple characters can be
printed using a single SBI call.

I miss a discussion of the conflicts that can arise if the configuration
of the serial console is changed by the caller.

Do we need an ecall that closes the SBI console to further access?

Usually, the serial port related code in M-mode firmware only uses
status and data registers so for most serial ports support the M-mode
firmware will adapt to serial port configuration changes.

In fact, this is why we never had a special SBI call for serial port
reconfiguration in legacy SBI v0.1 as well.

In case of virtualization, the serial port (or console) is emulated so
the special SBI call is not useful for virtualization.

The supervisor-mode software must set the shared memory area before
printing characters on the debug console. Also, all HARTs share the
same shared memory area so only one HART needs to set it at boot-time.

Isn't it M-mode software that has to program the MMU to allow all harts
in M-mode and S-mode access to the memory area? What is the S-mode
software to do about the memory area prior to calling
sbi_debug_console_set_area()?

Actually, it's the S-mode software which is voluntarily giving a portion of
its memory to be used as shared memory. The proposal only mandates
that whatever memory is selected by S-mode software should be a
regular cacheable memory (not IO memory). Also, if Svpbmt is available
then S-mode software should only use memory type 0 in the PTEs.

Function: Set Console Area (FID #0)
-----------------------------------

struct sbiret sbi_debug_console_set_area(unsigned long addr_div_by_4,
unsigned long size)

The console output is needed on the very start of the S-mode software,
before setting up anything.

Can we avoid this extra function?

Can we simply assume that the caller of sbi_debug_console_puts()
provides a physical address pointer to a memory area that is suitable?

Theoretically, we can avoid the extra function to set shared memory area
but it will complicate things in future when we have supervisor software
encrypting it's own memory (using special ISA support) because in this
case supervisor software will have to unprotect memory every time the
sbi_debug_console_puts() is called and protect it again after the call.

Currently this function is just a nop(). It is not needed in this
revision of the extension.

The function might be called repeatedly by different threads with
different values. How do you want to keep track of all of these
different areas?

The shared memory area in case of this SBI extension will be shared
across all HARTs so the SBI implementation will ensure atomicity
in setting/reading shared memory coordinates. This way multiple
HARTs can call the sbi_debug_console_set_area() but only the
last call will be in effect.

Following text in the proposal explains above:
"The supervisor-mode software must set the shared memory area before
printing characters on the debug console. Also, all HARTs share the
same shared memory area so only one HART needs to set it at boot-time."

Maybe this text is not giving a clear picture ?

Memory shared between different security realms will arise in many
different scenarios. As this is not console specific it should be in a
separate extension. That extension should be defined once we have
clarity about how security realms are managed.

In the case of this extension, the shared memory is global across all
HARTs and it mostly contains bytes to be printed. In case of steal
time accounting, the shared memory is separate for each HART and
it is a well-defined data structure.

Clearly, the usage of shared memory is extension specific.

The VirtIO devices are a good (and time tested) example of shared
memory based approaches. Over there as well, the shared memory
(i.e. various VirtIO rings) are setup by Guest and there is no central
pool of shared memory (i.e no dedicated VirtIO device managing
shared memory).

Similar to VirtIO world, we should let SBI extension define its own
shared memory usage and API.

Regards,
Anup

Set the shared memory area specified by `addr_div_by_2` and `size`

%s/addr_div_by_2/addr_div_by_4/

Okay, I will update.

parameters. The `addr_div_by_4` parameter is base address of the

%s/is base/is the base/

Okay, I will update.

shared memory area right shifted by 2 whereas `size` parameter is
the size of shared memory area in bytes.

Why shifting the address? I would prefer to keep it simple for the
caller. If the alignment is not suitable, return an error.

But why is an alignment needed here at all? And why 4 aligned?

For RV32 S-mode, the physical address space is 34bits wide but
"unsigned long" is 32bits wide. This is because Sv32 PTEs allow
34bits of PPN. In fact, even instructions such as HFENCE.GVMA
have this "address right shift by 2" requirement based on this rationale.

The shared memory area should be normal cacheable memory for the
supervisor-mode software. Also, the shared memory area is global
across all HARTs so SBI implementation must ensure atomicity in
setting the shared memory area.

Errors:
SBI_SUCCESS - Shared memory area set successfully.
SBI_ERR_INVALID_ADDRESS - The shared memory area pointed by
`addr_div_by_2` and `size` parameters
is not normal cacheable memory or not
accessible to supervisor-mode software.

Function: Console Puts (FID #1)
-------------------------------

struct sbiret sbi_debug_console_puts(unsigned long area_offset,
unsigned long num_chars)

I would prefer to simply pass a physical address pointer here with no
requirements on alignment. And no prior SBI call.

sbi_debug_console_set_area() might be called with different values by
different threads. An offset is ambiguous as it does not define to which
of the different shared areas it relates. Please, use a pointer.

Do we need num_chars? Are we expecting to provide binary output? Using
0x00 as terminator would be adequate in most cases.

Bare-metal tests (or assembly sources) can print sub-strings from
a large per-populated string in shared memory. Assuming that string
is always terminated by 0x00 in sbi_debug_console_puts() will break
this flexibility.

OK

What is the requirement on the console? Does it have to support 8bit
output to allow for UTF-8?

We need to clarify this. Suggestions ?

The platform specification would be the right place to require 8-bit
support for the console.

Do we make any assumptions about encoding?

Same as above, this needs more clarification. Suggestions ?

We should add to the platform specification that UTF-8 output is assumed
on the serial console.

I am of the opinion to keep such encoding related assumptions to be
minimal.

How would we handle a console set up to 7bit + parity if a character >
0x7f is sent?

I would consider this to be part of the clarification we add for encoding.

We should state if extra bits are ignored or the bytes are not send.
The easiest thing is to just ignore the extra bits. So let't state this
here.

Best regards

Heinrich

Print the string specified by `area_offset` and `num_chars` on
the debug console. The `area_offset` parameter is the start of
string in the shard memory area whereas `num_chars` parameter

%s/shard/shared/

Okay, I will update.

is the number of characters (or bytes) in the string.

This is a blocking SBI call and will only return after printing
all characters of the string.

Errors:
SBI_SUCCESS - Characters printed successfully.
SBI_ERR_INVALID_ADDRESS - The start of the string (i.e.
`area_offset`) or end of the string
(i.e. `area_offset + num_chars`) is
outside shared memory area.

There could be other reasons of failures:

* set up of the UART failed in OpenSBI
* no UART defined in the device-tree
* ...

So let us add SBI_ERR_FAILED to the list.

Okay, I will add.

Best regards

Heinrich

Regards,
Anup

On Thu, Jun 2, 2022 at 2:58 PM Heiko Stübner <heiko@...> wrote:

Am Donnerstag, 2. Juni 2022, 10:50:56 CEST schrieb Heiko Stübner:

Am Donnerstag, 2. Juni 2022, 10:47:58 CEST schrieb Anup Patel:

On Thu, Jun 2, 2022 at 12:02 AM Heiko Stübner <heiko@...> wrote:

Hi Anup,

Am Mittwoch, 1. Juni 2022, 18:17:32 CEST schrieb Anup Patel:

Hi All,

Below is the draft proposal for SBI Debug Console Extension.

Please review it and provide feedback.

Thanks,
Anup

Debug Console Extension (EID #0x4442434E "DBCN")
================================================

The debug console extension defines a generic mechanism for boot-time
early prints from supervisor-mode software which allows users to catch
boot-time issues in supervisor-mode software.

This extension replaces legacy console putchar (EID #0x01) extension
and it is better in following ways:
1) It follows the new calling convention defined for SBI v1.0
(or higher).
2) It is based on a shared memory area between SBI implementation
and supervisor-mode software so multiple characters can be
printed using a single SBI call.

The supervisor-mode software must set the shared memory area before
printing characters on the debug console. Also, all HARTs share the
same shared memory area so only one HART needs to set it at boot-time.

Function: Set Console Area (FID #0)
-----------------------------------

struct sbiret sbi_debug_console_set_area(unsigned long addr_div_by_4,
unsigned long size)

Set the shared memory area specified by `addr_div_by_2` and `size`

typo in the "div_by_2" (not 4 like below and in the function itself) ?

parameters. The `addr_div_by_4` parameter is base address of the
shared memory area right shifted by 2 whereas `size` parameter is
the size of shared memory area in bytes.

The shared memory area should be normal cacheable memory for the
supervisor-mode software. Also, the shared memory area is global
across all HARTs so SBI implementation must ensure atomicity in
setting the shared memory area.

Errors:
SBI_SUCCESS - Shared memory area set successfully.
SBI_ERR_INVALID_ADDRESS - The shared memory area pointed by
`addr_div_by_2` and `size` parameters
is not normal cacheable memory or not
accessible to supervisor-mode software.

Function: Console Puts (FID #1)
-------------------------------

struct sbiret sbi_debug_console_puts(unsigned long area_offset,
unsigned long num_chars)

Print the string specified by `area_offset` and `num_chars` on
the debug console. The `area_offset` parameter is the start of
string in the shard memory area whereas `num_chars` parameter
is the number of characters (or bytes) in the string.

This is a blocking SBI call and will only return after printing
all characters of the string.

Errors:
SBI_SUCCESS - Characters printed successfully.
SBI_ERR_INVALID_ADDRESS - The start of the string (i.e.
`area_offset`) or end of the string
(i.e. `area_offset + num_chars`) is
outside shared memory area.

This will vastly reduce the number of needed ecalls when outputting
characters, so this will probably improve performance quite a bit :-)


I guess I still would like to have an _additional_ single-character
putc call. As mentioned in the other thread [0], especially on consumer
hardware [where there is no elaborate debug infrastructure] this can
be a very handy debugging tool even in the earliest stages of a
booting kernel (both before relocation and even inside the startup
assembly).

I.e. just doing a
li a7, 1
li a6, 0
li a0, 36
ecall

in any kernel assembly will just output a "$" character right now, without
needing any preparation at all - same with using the current
sbi_console_putchar() directly in c-code.

This _can_ be very helpful in some cases, so I guess it would be nice
to keep such a functionality around also in the new spec.

You can easily create multiple pre-populated strings using ".asciiz" in
assembly sources. Just set the base address of pre-populated strings
once on boot hart and print from anywhere using usual 4-5 instruction
(similar to what posted above).

ok, sounds like a plan as well :-)

though, how does that relate to the time before MMU setup?

I.e. in response to Heinrich's mail you talk about svpbmt, so I guess you
expect virtual memory there, so what is the expected value-type before
the mmu is setup in S-mode ?

The memory type should be 0 (i.e. PMA) for the shared memory between
SBI implementation and supervisor software. Before MMU setup, the
memory type is by default 0 (i.e. PMA) so we don't have to mandate
any memory type for MMU disabled case.

We only have issue on systems with Svpmbt where supervisor software
can potentially map the shared memory as non-cacheable or IO (memory
type != 0) using PTE memory type bits.

In addition to above, a SBI implementation must ensure that the shared
memory address provided by supervisor software is a regular memory
(not MMIO device). This can be easily achieved in OpenSBI, KVM RISC-V,
and various hypervisors.

Regards,
Anup

[0] http://lists.infradead.org/pipermail/opensbi/2022-June/002796.html

Regards,
Anup

On Thu, Jun 2, 2022 at 10:08 AM Xiang W <wxjstz@...> wrote:

在 2022-06-01星期三的 21:47 +0530，Anup Patel写道：

Hi All,

Below is the draft proposal for SBI Debug Console Extension.

Please review it and provide feedback.

Thanks,
Anup

Debug Console Extension (EID #0x4442434E "DBCN")
================================================

The debug console extension defines a generic mechanism for boot-time
early prints from supervisor-mode software which allows users to catch
boot-time issues in supervisor-mode software.

This extension replaces legacy console putchar (EID #0x01) extension
and it is better in following ways:
1) It follows the new calling convention defined for SBI v1.0
(or higher).
2) It is based on a shared memory area between SBI implementation
and supervisor-mode software so multiple characters can be
printed using a single SBI call.

The supervisor-mode software must set the shared memory area before
printing characters on the debug console. Also, all HARTs share the
same shared memory area so only one HART needs to set it at boot-time.

Function: Set Console Area (FID #0)
-----------------------------------

struct sbiret sbi_debug_console_set_area(unsigned long addr_div_by_4,
unsigned long size)

Set the shared memory area specified by `addr_div_by_2` and `size`
parameters. The `addr_div_by_4` parameter is base address of the
shared memory area right shifted by 2 whereas `size` parameter is
the size of shared memory area in bytes.

The shared memory area should be normal cacheable memory for the
supervisor-mode software. Also, the shared memory area is global
across all HARTs so SBI implementation must ensure atomicity in
setting the shared memory area.

Errors:
SBI_SUCCESS - Shared memory area set successfully.
SBI_ERR_INVALID_ADDRESS - The shared memory area pointed by
`addr_div_by_2` and `size` parameters
is not normal cacheable memory or not
accessible to supervisor-mode software.

Shared memory can be a single extension. The three interfaces are as follows

/* The supervisor hands a piece of physical memory to sbi for shared memory */
struct sbiret sbi_shared_memory_extrend(unsigned long addr, unsigned long size);


/* The supervisor applies for a piece of physical memory from sbi */
struct sbiret sbi_shared_memory_alloc(unsigned long size);


/* The supervisor notifies sbi to release the requested memory */
struct sbiret sbi_shared_memory_free(unsigned long addr);

Clearly, if we have a separate SBI extension to manage shared memory
then we will end-up with such memory management calls. Memory allocators
are generally hard to get it right and this also adds lot of bookkeeping and
state management in SBI implementations (e.g. OpenSBI, KVM RISC-V,
and various hypervisors).

Further, some of the SBI extensions (such as Steal Time Accounting) will
have separate shared memory for each HART whereas some (such as
Debug Console) will have global shared memory for all HARTs.

For clean and modular SBI implementations, I would recommend that
each SBI extension define its own shared memory setup API.

Regards,
Anup

Function: Console Puts (FID #1)
-------------------------------

struct sbiret sbi_debug_console_puts(unsigned long area_offset,
unsigned long num_chars)

Print the string specified by `area_offset` and `num_chars` on
the debug console. The `area_offset` parameter is the start of
string in the shard memory area whereas `num_chars` parameter
is the number of characters (or bytes) in the string.

This is a blocking SBI call and will only return after printing
all characters of the string.

Errors:
SBI_SUCCESS - Characters printed successfully.
SBI_ERR_INVALID_ADDRESS - The start of the string (i.e.
`area_offset`) or end of the string
(i.e. `area_offset + num_chars`) is
outside shared memory area.

Hi Anup,

From: sig-hypervisors@... <sig-hypervisors@...> On Behalf Of Anup Patel via
lists.riscv.org
Subject: [sig-hypervisors] SBI Debug Console Extension Proposal (Draft v1)

Below is the draft proposal for SBI Debug Console Extension.

Here are my thoughts:

* Guest memory access: I think this would be the first SBI extension to require access to
guest memory. This needs to be considered carefully, but I think the higher bandwidth afforded by
the interface is useful enough to allow this.

* API:
* Currently, only a write interface is provided. It would be much better to have a
read/write interface.

Benefits of this would be to allow a hypervisor to control an OS, e.g., for testing purposes
or to automate installation tasks. Inter-guest communication could also be realized
via such an interface.

* As the relationship between SBI and OS is the same as OS and user process, an interface
in the style of e.g., Unix IO is possible.

* Global shared memory buffer design and alternatives:

The API should have per read() or write() parameters for buffer address and length.
This makes it easy for different parts of the OS kernel to output strings directly,
without requiring thread synchronization of the shared buffer used by the present proposal.
A single buffer (i.e., shared memory) will in most SW architectures require an
extra copying of the formatted output string into the shared memory region, which
would be avoided using per read()/write() call parameters.
The exact same argument applies to the SBI implementation: a multi-hart machine
utilizing a single shared memory block to communicate with S-mode software
will require m-mode thread synchronization when accessing the block.
Having only a single shared memory block will lead to scaling problems
on high hart count machines.

I see no advantages of the proposed design to dedicate a block of memory to
I/O in advance (and no modern API does this).
The SBI implementation will still need to be prepared to handle
access faults on each read and write call, there is no amortization of one-time costs
regarding address translation or permission checking that I see. In the H-extension,
the guest memory access instructions are as effective as possible, for M-mode code,
I'm sure that efficient access to S-mode virtual addresses is possible as well.
(Or the convention can be made that addresses are specified as guest-physical,
which should be avoided if efficient alternatives are available in RISC-V.
Note that the draft proposal does not speak of this at all).

* Multiple device support: a parameter should be allocated to allow the OS to
select which of possibly several output devices to utilize, like the "file
descriptor" parameter of the POSIX read()/write() interface.

This raises the question of binding these file descriptors to physical devices,
but a start could be made in analogy to Unix (again), where devices 0, 1, 2 are
assigned to standard in, standard out, and standard error, and in many cases
will be attached to the same physical device, and that are pre-opened when the
OS starts. As in Unix, the semantics of these would be roughly defined,
but an OS could write out its boot logging strings to 2, and drive an interactive
console process (such as a shell) from 0 and 1.

The question of binding string-valued names to file descriptor (open()) and
closing them, etc., could be deferred to a later date and possibly
be implementation defined.

* Asynchronicity and flow control: Unix solves this (poorly) via select()
or the SIGIO signal. If SBI introduces an interface for this, this will need to
co-exist with the OS scheduling that is largely driven by the interrupt controller,
which is a hard thing to do.

I think that the SBI interface should
be best effort, copying all data from the client immediately (as Unix does):
when buffer space in the SBI implementation is exhausted,
it should return short counts; if the SBI client is prepared to wait, a blocking
flush call could be added, which returns when some measure space is
available in the SBI implementation's output buffers (or when input is available).

The existing Unix interfaces for this can again be used as a guide; they
should be mature enough to cover all relevant cases.

For highest performance, an S-mode interrupt could be synthesized by the
SBI implementation when buffer space is available.

With best regards,
Konrad Schwarz

Siemens AG
T CED SES-DE
Otto-Hahn-Ring 6
81739 Munich, Germany
Phone: +49 (89) 7805-22579
Fax.: +49 (89) 636-33045
Mobile: +49 (1522) 8864636
mailto:konrad.schwarz@...
www.siemens.com

Siemens Aktiengesellschaft: Chairman of the Supervisory Board: Jim Hagemann Snabe; Managing Board: Roland Busch, Chairman, President and Chief Executive Officer; Cedrik Neike, Matthias Rebellius, Ralf P. Thomas, Judith Wiese; Registered offices: Berlin and Munich, Germany; Commercial registries: Berlin-Charlottenburg, HRB 12300, Munich, HRB 6684; WEEE-Reg.-No. DE 23691322

Am Donnerstag, 2. Juni 2022, 10:50:56 CEST schrieb Heiko Stübner:

Am Donnerstag, 2. Juni 2022, 10:47:58 CEST schrieb Anup Patel:

On Thu, Jun 2, 2022 at 12:02 AM Heiko Stübner <heiko@...> wrote:

Hi Anup,

Am Mittwoch, 1. Juni 2022, 18:17:32 CEST schrieb Anup Patel:

Hi All,

Below is the draft proposal for SBI Debug Console Extension.

Please review it and provide feedback.

Thanks,
Anup

Debug Console Extension (EID #0x4442434E "DBCN")
================================================

The debug console extension defines a generic mechanism for boot-time
early prints from supervisor-mode software which allows users to catch
boot-time issues in supervisor-mode software.

This extension replaces legacy console putchar (EID #0x01) extension
and it is better in following ways:
1) It follows the new calling convention defined for SBI v1.0
(or higher).
2) It is based on a shared memory area between SBI implementation
and supervisor-mode software so multiple characters can be
printed using a single SBI call.

The supervisor-mode software must set the shared memory area before
printing characters on the debug console. Also, all HARTs share the
same shared memory area so only one HART needs to set it at boot-time.

Function: Set Console Area (FID #0)
-----------------------------------

struct sbiret sbi_debug_console_set_area(unsigned long addr_div_by_4,
unsigned long size)

Set the shared memory area specified by `addr_div_by_2` and `size`

typo in the "div_by_2" (not 4 like below and in the function itself) ?

parameters. The `addr_div_by_4` parameter is base address of the
shared memory area right shifted by 2 whereas `size` parameter is
the size of shared memory area in bytes.

The shared memory area should be normal cacheable memory for the
supervisor-mode software. Also, the shared memory area is global
across all HARTs so SBI implementation must ensure atomicity in
setting the shared memory area.

Errors:
SBI_SUCCESS - Shared memory area set successfully.
SBI_ERR_INVALID_ADDRESS - The shared memory area pointed by
`addr_div_by_2` and `size` parameters
is not normal cacheable memory or not
accessible to supervisor-mode software.

Function: Console Puts (FID #1)
-------------------------------

struct sbiret sbi_debug_console_puts(unsigned long area_offset,
unsigned long num_chars)

Print the string specified by `area_offset` and `num_chars` on
the debug console. The `area_offset` parameter is the start of
string in the shard memory area whereas `num_chars` parameter
is the number of characters (or bytes) in the string.

This is a blocking SBI call and will only return after printing
all characters of the string.

Errors:
SBI_SUCCESS - Characters printed successfully.
SBI_ERR_INVALID_ADDRESS - The start of the string (i.e.
`area_offset`) or end of the string
(i.e. `area_offset + num_chars`) is
outside shared memory area.

This will vastly reduce the number of needed ecalls when outputting
characters, so this will probably improve performance quite a bit :-)


I guess I still would like to have an _additional_ single-character
putc call. As mentioned in the other thread [0], especially on consumer
hardware [where there is no elaborate debug infrastructure] this can
be a very handy debugging tool even in the earliest stages of a
booting kernel (both before relocation and even inside the startup
assembly).

I.e. just doing a
li a7, 1
li a6, 0
li a0, 36
ecall

in any kernel assembly will just output a "$" character right now, without
needing any preparation at all - same with using the current
sbi_console_putchar() directly in c-code.

This _can_ be very helpful in some cases, so I guess it would be nice
to keep such a functionality around also in the new spec.

You can easily create multiple pre-populated strings using ".asciiz" in
assembly sources. Just set the base address of pre-populated strings
once on boot hart and print from anywhere using usual 4-5 instruction
(similar to what posted above).

ok, sounds like a plan as well :-)

though, how does that relate to the time before MMU setup?

I.e. in response to Heinrich's mail you talk about svpbmt, so I guess you
expect virtual memory there, so what is the expected value-type before
the mmu is setup in S-mode ?

[0] http://lists.infradead.org/pipermail/opensbi/2022-June/002796.html

Regards,
Anup