On Fri, 2022-06-03 at 07:49 +0000, Schwarz, Konrad wrote:

From:
sig-hypervisors@... <sig-hypervisors@...>
On Behalf Of Anup Patel via
lists.riscv.org
Sent: Friday, June 3, 2022 5:01

We have legacy SBI v0.1 calls where most of them are replaced by
newer SBI v0.2 (or higher) calls. Only the SBI v0.1 putchar() does
not
have a replacement. This putchar() was being used in various cases
for early prints from OSes and bootloaders.

The SBI debug console extension is an attempt to replace the legacy
SBI v0.1 putchar(). The use of shared memory in this proposal is
motivated by the need to allow users to print multiple characters
in
one SBI call.

As I pointed out in an earlier message, the design using a shared
memory block
is a hindrance.  Instead, each call should provide a pointer to the
(virtual)
address of the character buffer.

I agree

With the current design there is no way for the supervisor software to
reclaim the memory once it has shared it with OpenSBI. We can add a
close() call, but it seems simpler and safer to instead just pass an
address and size to print.

The legacy SBI v0.1 also had a getchar() call which is deprecated
and
does not need any newer SBI call replacing it because it is
expected
all platforms (including virtual platforms emulated by hypervisors)
will
have a proper interrupt driver console for supervisor software. The
proposed SBI debug console extension only tries to provide a
solution
for early prints.

Consider the case where we have more VMs than physical UARTs -- this
will actually be the norm in most deployments.

In this case you should use virtIO or some other more powerful
interface

(To counter the argument that the hypervisor can provide virtual
UARTs
for its guests, note that a dedicated para-virtualized interface is
much more
efficient than trapping individual memory accesses to simulated HW
registers).

Agreed, but tha seems up to the Hypervisor to implement an effient
virtual UART implementation. For example the Xen HYPERVISOR_console_io
that Stefano mentioned earlier. Plus then we can leverage existing
implementations insted of inveting a another virtual serial device.

By making the interface a bit richer, as discussed in my earlier
proposal,
a hypervisor can cover a much larger set of use cases.

Which we also then need to support in M-mode firmware. The more complex
the firmware becomes the more exposed it is.

Alistair

--
Konrad

From: sig-hypervisors@... <sig-hypervisors@...> On Behalf Of Anup Patel via
lists.riscv.org
Sent: Friday, June 3, 2022 5:01

We have legacy SBI v0.1 calls where most of them are replaced by
newer SBI v0.2 (or higher) calls. Only the SBI v0.1 putchar() does not
have a replacement. This putchar() was being used in various cases
for early prints from OSes and bootloaders.

The SBI debug console extension is an attempt to replace the legacy
SBI v0.1 putchar(). The use of shared memory in this proposal is
motivated by the need to allow users to print multiple characters in
one SBI call.

As I pointed out in an earlier message, the design using a shared memory block
is a hindrance. Instead, each call should provide a pointer to the (virtual)
address of the character buffer.

The legacy SBI v0.1 also had a getchar() call which is deprecated and
does not need any newer SBI call replacing it because it is expected
all platforms (including virtual platforms emulated by hypervisors) will
have a proper interrupt driver console for supervisor software. The
proposed SBI debug console extension only tries to provide a solution
for early prints.

Consider the case where we have more VMs than physical UARTs -- this
will actually be the norm in most deployments.

(To counter the argument that the hypervisor can provide virtual UARTs
for its guests, note that a dedicated para-virtualized interface is much more
efficient than trapping individual memory accesses to simulated HW registers).

By making the interface a bit richer, as discussed in my earlier proposal,
a hypervisor can cover a much larger set of use cases.


--
Konrad

Hi Stefano,

On Fri, Jun 3, 2022 at 2:35 AM Stefano Stabellini
<stefano.stabellini@...> wrote:

On Thu, 2 Jun 2022, Greg Favor wrote:

On Thu, Jun 2, 2022 at 1:29 PM Stefano Stabellini <stefano.stabellini@...> wrote:
On Thu, 2 Jun 2022, Vedvyas Shanbhogue via lists.riscv.org wrote:
> Of course a console is needed. But I was questioning the need for something much
> more elaborate than a putchar/getchar interface. I understand its needed to port
> the hypervisor but I undersatnd it would be needed for very early parts of the
> hypervisor where it does not even have the ability to master a uart on its own.

Yeah, I think you are right that it is not a must-have feature but a
nice-to-have feature.


Even though we all (me included) tend to think that slowing down something like this will be inconsequential to the time to boot and/or who
cares how fast or slow is boot time, in the past I have found that to be a risky presumption in certain production environments where boot
time matters (especially with frequent standing up of VMs from scratch). So a worthwhile sanity check question would be 1) what percentage
of early boot time is spent writing to the console with per-string SBI calls, and 2) by what order of magnitude does that percentage
inflate due to SBI calls for every individual character.

The numbers you asked will help but I just wanted to add that in my
opinion a per-character putchar interface is not suitable for production
due to the reasons you mentioned. I would compile it out in non-DEBUG
builds.

On the other hand the per-string print interface might be usable in
production, even not just early boot. However, of course, a proper UART
driver would be even better.

This is why I think it is nice to have: it expands the potential uses of
the SBI console interface, and it is simple and easy to use. At the same
time it is not a must-have because you could get away with just putchat
in early boot for DEBUG builds, and nothing + UART driver in non-DEBUG
builds.

That is why I am in favor of this addition, although I recognize it is
not critical.

We have legacy SBI v0.1 calls where most of them are replaced by
newer SBI v0.2 (or higher) calls. Only the SBI v0.1 putchar() does not
have a replacement. This putchar() was being used in various cases
for early prints from OSes and bootloaders.

The SBI debug console extension is an attempt to replace the legacy
SBI v0.1 putchar(). The use of shared memory in this proposal is
motivated by the need to allow users to print multiple characters in
one SBI call.

The legacy SBI v0.1 also had a getchar() call which is deprecated and
does not need any newer SBI call replacing it because it is expected
all platforms (including virtual platforms emulated by hypervisors) will
have a proper interrupt driver console for supervisor software. The
proposed SBI debug console extension only tries to provide a solution
for early prints.

Regards,
Anup

On Thu, 2 Jun 2022, Greg Favor wrote:

On Thu, Jun 2, 2022 at 1:29 PM Stefano Stabellini <stefano.stabellini@...> wrote:
On Thu, 2 Jun 2022, Vedvyas Shanbhogue via lists.riscv.org wrote:
> Of course a console is needed. But I was questioning the need for something much
> more elaborate than a putchar/getchar interface. I understand its needed to port
> the hypervisor but I undersatnd it would be needed for very early parts of the
> hypervisor where it does not even have the ability to master a uart on its own.

Yeah, I think you are right that it is not a must-have feature but a
nice-to-have feature.


Even though we all (me included) tend to think that slowing down something like this will be inconsequential to the time to boot and/or who
cares how fast or slow is boot time, in the past I have found that to be a risky presumption in certain production environments where boot
time matters (especially with frequent standing up of VMs from scratch).  So a worthwhile sanity check question would be 1) what percentage
of early boot time is spent writing to the console with per-string SBI calls, and 2) by what order of magnitude does that percentage
inflate due to SBI calls for every individual character.

The numbers you asked will help but I just wanted to add that in my
opinion a per-character putchar interface is not suitable for production
due to the reasons you mentioned. I would compile it out in non-DEBUG
builds.

On the other hand the per-string print interface might be usable in
production, even not just early boot. However, of course, a proper UART
driver would be even better.

This is why I think it is nice to have: it expands the potential uses of
the SBI console interface, and it is simple and easy to use. At the same
time it is not a must-have because you could get away with just putchat
in early boot for DEBUG builds, and nothing + UART driver in non-DEBUG
builds.

That is why I am in favor of this addition, although I recognize it is
not critical.

On Thu, Jun 02, 2022 at 01:50:01PM -0700, Greg Favor wrote:

On Thu, Jun 2, 2022 at 1:29 PM Stefano Stabellini <
stefano.stabellini@...> wrote:

On Thu, 2 Jun 2022, Vedvyas Shanbhogue via lists.riscv.org wrote:

Of course a console is needed. But I was questioning the need for

something much

more elaborate than a putchar/getchar interface. I understand its needed

to port

the hypervisor but I undersatnd it would be needed for very early parts

of the

hypervisor where it does not even have the ability to master a uart on

its own.

Yeah, I think you are right that it is not a must-have feature but a
nice-to-have feature.

Even though we all (me included) tend to think that slowing down something
like this will be inconsequential to the time to boot and/or who cares how
fast or slow is boot time, in the past I have found that to be a risky
presumption in certain production environments where boot time matters
(especially with frequent standing up of VMs from scratch). So a
worthwhile sanity check question would be 1) what percentage of early boot
time is spent writing to the console with per-string SBI calls, and 2) by
what order of magnitude does that percentage inflate due to SBI calls for
every individual character.

I think the extension was proposed to be a Debug console extension.

In a production environment if 10's of VMs were booting having them emit
to the physical console - where all of those outputs will get interspersed
seems to be not so useful. In a production environment would we expect the
VMM to provide a console service and not need SBI calls into M-mode firmare
to emit VM output to a physical console?

I was thinking this was about early boot debug and not so much about needing
a console at early boot. In most production deployments there may not be someone sitting at a terminal watching prints and so it may make more sense
for VM console outputs to be logged into a persistent file somewhere for example.
Or be held in the VM itself - e.g. a kernel ring buffer that someone may want
to display using a tool like dmesg when needed but otherwise does not get emitted chattily to a physical console.

Hope I am understanding this right.

regards
ved

On Thu, 2 Jun 2022, Vedvyas Shanbhogue via lists.riscv.org wrote:

Of course, if a hypervisor is already available for the board, then it
would be just as easy to use a paravirtualized interface, e.g. Xen's
HYPERVISOR_console_io hypercall. But somebody has to port the
hypervisor first :-)

Of course a console is needed. But I was questioning the need for something
much
more elaborate than a putchar/getchar interface. I understand its needed to
port
the hypervisor but I undersatnd it would be needed for very early parts of the
hypervisor where it does not even have the ability to master a uart on its
own.

Yeah, I think you are right that it is not a must-have feature but a
nice-to-have feature.

Of course, if a hypervisor is already available for the board, then it
would be just as easy to use a paravirtualized interface, e.g. Xen's
HYPERVISOR_console_io hypercall. But somebody has to port the
hypervisor first :-)

Of course a console is needed. But I was questioning the need for something much
more elaborate than a putchar/getchar interface. I understand its needed to port
the hypervisor but I undersatnd it would be needed for very early parts of the hypervisor where it does not even have the ability to master a uart on its own.

regards
ved

On Thu, 2 Jun 2022, Vedvyas Shanbhogue via lists.riscv.org wrote:

Based on discussion it did not seem like it needs to be much fancier than
this as this is for early OS/VMM code till it has enough functionality to
directly interact with a uart.

The goal of the shared memory based SBI call for early prints is to
minimize the number of traps which in-turn helps virtualization to
drastically reduce boot-time.

Understand that better now. But if that is the main motivation then
I am not understanding why we would want to push all of this into
M-mode firmware vs. defining a set of standardized pv-ops to be used
by guest OSes.

That is because it is useful to have debug console output when porting a
hypervisor or baremetal code to a new board.

Of course, if a hypervisor is already available for the board, then it
would be just as easy to use a paravirtualized interface, e.g. Xen's
HYPERVISOR_console_io hypercall. But somebody has to port the
hypervisor first :-)