Date
1 - 3 of 3
[PATCH v3 4/6] Re-write the interrupts and timer section
|
On 06/08/21, 1:17 PM, "Heinrich Schuchardt" <heinrich.schuchardt@...> wrote:
On 8/6/21 8:19 AM, Anup Patel wrote: > We re-write the interrupts and timer section to align the > interrupts and timer table. We also add more fine grained > requirements for AIA. > > Signed-off-by: Anup Patel <anup.patel@...> > --- > Makefile | 4 +- > riscv-platform-spec.adoc | 323 ++++++++++++++++++++++----------------- > riscv-platform-spec.yml | 8 + > 3 files changed, 193 insertions(+), 142 deletions(-) > create mode 100644 riscv-platform-spec.yml > > diff --git a/Makefile b/Makefile > index 48e8fe9..ff8fbf6 100644 > --- a/Makefile > +++ b/Makefile > @@ -22,8 +22,8 @@ $(PLATFORM_SPEC).md: $(PLATFORM_SPEC).xml > $(PLATFORM_SPEC).xml: $(PLATFORM_SPEC).adoc > $(ASCIIDOCTOR) -d book -b docbook $< > > -$(PLATFORM_SPEC).pdf: $(PLATFORM_SPEC).adoc $(IMAGES) > - $(ASCIIDOCTOR) -d book -r asciidoctor-pdf -b pdf $< > +$(PLATFORM_SPEC).pdf: $(PLATFORM_SPEC).adoc $(IMAGES) $(PLATFORM_SPEC).yml > + $(ASCIIDOCTOR) -d book -r asciidoctor-pdf -a pdf-style=$(PLATFORM_SPEC).yml -b pdf $< > > $(PLATFORM_SPEC).html: $(PLATFORM_SPEC).adoc $(IMAGES) > $(ASCIIDOCTOR) -d book -b html $< > diff --git a/riscv-platform-spec.adoc b/riscv-platform-spec.adoc > index cffb71c..3994ef3 100644 > --- a/riscv-platform-spec.adoc > +++ b/riscv-platform-spec.adoc > @@ -13,6 +13,7 @@ > :doctype: book > :sectnums: > :sectnumlevels: 5 > +:xrefstyle: short > :toc: macro > :toclevels: 5 > > @@ -235,114 +236,173 @@ each must be 1 > The default should allow code that's sensitive to these requirements to be > debugged. > > -==== Interrupt Controller, Software Interrupt, and Timer Requirements > -In the following requirements, > -https://github.com/riscv/riscv-aia[*AIA*] refers to the Advanced Interrupt Architecture, https://github.com/riscv/riscv-aclint/blob/main/riscv-aclint.adoc[*ACLINT*] > -refers to the Advanced *CLINT*. AIA comprises two separate components: `IMSICs` and `APLICs`. > -If supported, there is an `IMSIC` device associated with each hart. > -If supported, *APLIC* devices are global to all harts, and there may be one or > -multiple in a system. *ACLINT* comprises three separate components: `MTIMER` for > -Timer support, and `MSWI` and `SSWI` for Machine-level and Supervisor-level > -Software Interrupt (IPI) support. + > -https://github.com/riscv/riscv-plic-spec/blob/master/riscv-plic.adoc[*PLIC*] > -refers to the legacy Platform-Level Interrupt Controller that provides > -facilities to route external interrupts to a hart context with a given privilege > -mode. The number of non-local interrupt sources supported by PLIC and how does > -each of them connect to the hart context is PLIC core implementation-specific. + > -*CLINT* is a legacy Core-Local Interrupt Controller that is a compatible subset of > -ACLINT which provides facilities to trigger Software (IPI) and Timer interrupts to > -hart. > +==== Interrupts and Timer > + > +===== Timer support > + > +* One or more ACLINT MTIMER devices are required for the OS-A platform > +* All MTIME counters must be 64-bit counters with a 10ns resolution (i.e. > + an increment by 1 represents 10 ns). > +* The MTIME update frequency (i.e. hardware clock) must be between 10 MHz > + and 100 MHz, and updates must be strictly monotonic. > +* Platforms must provide a way to discover MTIME resolution using appropriate > + hardware discovery mechanism: > +** Platforms that use DT for hardware discovery are required to advertise the > + timebase to the operating systems via the `timebase-frequency` property of > + the "/cpus" node > +footnote:[https://elixir.bootlin.com/linux/latest/source/Documentation/devicetree/bindings/riscv/cpus.yaml]. > + > +[sidebar] > +-- > +[underline]*_Implementation Note_* > > -.The following table summarizes what features are supported for four classes of OS/A platforms. > -[width="100%",cols="^,^,^,^,^,^,^,^,^,^,^,^,^"] > -|======= > -.2+|*OS-A Platform* 3+|*MSIs* 3+|*Wired Interrupts* 3+|*IPIs* 3+|*Timer* > -|M-level|S-level|VS-level|M-level|S-level|VS-level|M-level|S-level|VS-level|M-level|S-level|VS-level > -|Existing|NA|NA|NA|PLIC|PLIC|PLIC + > -_(Trap and emulate)_|MSWI + > -https://github.com/riscv/riscv-aclint/blob/main/riscv-aclint.adoc[`*ACLINT*`]| > -SBI IPI + > -https://github.com/riscv/riscv-sbi-doc[`*SBI*`]|SBI IPI + > -https://github.com/riscv/riscv-sbi-doc[`*SBI*`]|MTIMER + > -https://github.com/riscv/riscv-aclint/blob/main/riscv-aclint.adoc[`*ACLINT*`]|SBI Timer + > -https://github.com/riscv/riscv-sbi-doc[`*SBI*`]|SBI Timer + > -https://github.com/riscv/riscv-sbi-doc[`*SBI*`] > -|Only Wired IRQs|NA|NA|NA|APLIC M-level + > -https://github.com/riscv/riscv-aia[`*AIA*`]|APLIC S-level + > -https://github.com/riscv/riscv-aia[`*AIA*`]|APLIC S-level + > -_(Trap and emulate)_ + > -https://github.com/riscv/riscv-aia[`*AIA*`]|MSWI + > -https://github.com/riscv/riscv-aclint/blob/main/riscv-aclint.adoc[`*ACLINT*`]| > -SSWI + > -https://github.com/riscv/riscv-aclint/blob/main/riscv-aclint.adoc[`*ACLINT*`]| > -SBI IPI + > -https://github.com/riscv/riscv-sbi-doc[`*SBI*`]|MTIMER + > -https://github.com/riscv/riscv-aclint/blob/main/riscv-aclint.adoc[`*ACLINT*`]| > -Priv Sstc + > -https://github.com/riscv/riscv-isa-manual/releases[`*Priv Spec*`]| > -Priv Sstc + > -https://github.com/riscv/riscv-isa-manual/releases[`*Priv Spec*`] > -|MSIs and Wired IRQs|IMSIC M-level + > -https://github.com/riscv/riscv-aia[`*AIA*`]|IMSIC S-level + > -https://github.com/riscv/riscv-aia[`*AIA*`]|APLIC S-level + > -_(Trap and emulate)_ + > -https://github.com/riscv/riscv-aia[`*AIA*`]|APLIC M-level + > -https://github.com/riscv/riscv-aia[`*AIA*`]|APLIC S-level + > -https://github.com/riscv/riscv-aia[`*AIA*`]|APLIC S-level + > -_(Trap and emulate)_ + > -https://github.com/riscv/riscv-aia[`*AIA*`]|IMSIC M-level + > -https://github.com/riscv/riscv-aia[`*AIA*`]|IMSIC S-level + > -https://github.com/riscv/riscv-aia[`*AIA*`]|SBI IPI + > -https://github.com/riscv/riscv-sbi-doc[`*SBI*`]|MTIMER + > -https://github.com/riscv/riscv-aclint/blob/main/riscv-aclint.adoc[`*ACLINT*`]| > -Priv Sstc + > -https://github.com/riscv/riscv-isa-manual/releases[`*Priv Spec*`]| > -Priv Sstc + > -https://github.com/riscv/riscv-isa-manual/releases[`*Priv Spec*`] > -|MSIs, Virtual MSIs and Wired IRQs|IMSIC M-level + > -https://github.com/riscv/riscv-aia[`*AIA*`]|IMSIC S-level + > -https://github.com/riscv/riscv-aia[`*AIA*`]|APLIC VS-level + > -https://github.com/riscv/riscv-aia[`*AIA*`]|APLIC M-level + > -https://github.com/riscv/riscv-aia[`*AIA*`]|APLIC S-level + > -https://github.com/riscv/riscv-aia[`*AIA*`]|APLIC S-level + > -_(Trap and emulate)_ + > -https://github.com/riscv/riscv-aia[`*AIA*`]|IMSIC M-level + > -https://github.com/riscv/riscv-aia[`*AIA*`]|IMSIC S-level + > -https://github.com/riscv/riscv-aia[`*AIA*`]|IMSIC VS-level + > -https://github.com/riscv/riscv-aia[`*AIA*`]|MTIMER + > -https://github.com/riscv/riscv-aclint/blob/main/riscv-aclint.adoc[`*ACLINT*`]| > -Priv Sstc + > -https://github.com/riscv/riscv-isa-manual/releases[`*Priv Spec*`]| > -Priv Sstc + > -https://github.com/riscv/riscv-isa-manual/releases[`*Priv Spec*`] > -|======= > - > -* For Timer support, one or more ACLINT MTIMER devices are Required for OS-A platform. > -One MTIMER may be used for all harts, or multiple MTIMERs may be used with > -multiple topological groups of harts. The base address of MTIMER memory map registers > -is platform implementation-specific, however, the format of MTIMER operation parameters > -(`mtime` and `mtimecmp` registers) must be compliant with > -https://github.com/riscv/riscv-aclint/blob/main/riscv-aclint.adoc#21-register-map[ACLINT MTIMER Register Map] > - > -* For Interrupt Controller and Software Interrupt support, one of the following three > -choices below is Required > - ** PLIC plus one or more ACLINT MSWI devices - DEPRECATED > - *** One MSWI may be used for all harts, or multiple MSWIs may be used with > -multiple topological groups of harts > - *** Only wired interrupts and M-mode IPIs are supported > - *** Virtualization is not supported > - *** This compatibly supports legacy PLIC + CLINT designs > - ** One or more AIA APLIC devices plus one or more pairs of ACLINT MSWI and ACLINT SSWI devices > - *** One MSWI/SSWI pair may be used for all harts, or multiple MSWI/SSWI > -pairs may be used with multiple topological groups of harts > - *** Only wired interrupts are supported > - *** Both M-mode and S-mode IPIs are supported > - *** Virtualization is not supported > - ** Zero, one, or more AIA APLIC devices plus per-hart AIA IMSIC devices > - *** Both wired and MSI external interrupts are supported > - *** Both M-mode and S-mode IPIs are supported via IMSICs > - *** Virtualization is supported > - *** Zero APLICs if there are no wired interrupts and only MSIs > +For example, if `timebase-frequency` is 100 MHz and the MTIME update > +frequency (i.e. hardware clock) is 25 MHz then the MTIME counter would > +increment by 4 upon every hardware clock tick. > +-- > + > +===== Interrupts support > + > +The OS/A platorms can comply with one of the following four categories for > +interrupts support: > + > +* *Legacy IRQs - DEPRECATED* > +** One or more PLIC devices to support wired interrupts > +** One or more ACLINT MSWI devices to support M-mode software interrupts > +** Software interrupts for S-mode and VS-mode supported using the > + SBI IPI extension > +** Compatibile with legacy platforms having PLIC plus CLINT devices > +** MSI external interrupts not supported > +** MSI virtualization is not supported > +* *Only Wired IRQs* > +** One or more AIA APLIC devices to support wired interrupts > +** One or more ACLINT MSWI devices to support M-mode software interrupts > +** One or more ACLINT SSWI devices to support HS-mode software interrupts > +** Software interrupts for VS-mode supported using the SBI IPI extension > +** MSI external interrupts not supported > +** MSI virtualization is not supported > +* *MSIs and Wired IRQs* > +** AIA local interrupt CSRs must be supported by each hart > +*** `mvien` and `mvip` CSRs must be implemented > +*** `hvien` and `hvip` CSRs must be implemented if H-extension is implemented > +*** `siselect` CSR must support holding a 9-bit value > +*** `vsiselect` CSR must support holding a 9-bit value if H-extension > + is implemented > +** Per-hart AIA IMSIC devices must support MSIs for M-mode and HS-mode > +*** IPRIOLEN = 6 to 8 > +*** MIFMAX = SIFMAX = 63 to 255 > +*** Must support at least 63 distinct interrupt identities > +*** Must implement `seteipnum_le` memory-mapped register > +** One, or more AIA APLIC devices to support wired interrupts > +*** EIID and IID fields must be 6 to 8 bits wide matching the number of > + interrrupt identities supported by AIA IMSIC > +** Software interrupts for M-mode and HS-mode supported using AIA IMSIC devices > +** Software interrupts for VS-mode supported using the SBI IPI extension > +** MSI virtualization is not supported > +* *MSIs, Virtual MSIs, and Wired IRQs* > +** The H-extension must be implemeted by each hart to support virtual MSIs > +*** GEILEN must be 1 or more > +** AIA local interrupt CSRs must be supported by each hart > +*** `mvien` and `mvip` CSRs must be implemented > +*** `hvien` and `hvip` CSRs must be implemented > +*** `siselect` CSR must support holding a 9-bit value > +*** `vsiselect` CSR must support holding a 9-bit value > +** Per-hart AIA IMSIC devices to support MSIs for M-mode, HS-mode and VS-mode > +*** IPRIOLEN = 6 to 8 > +*** MIFMAX = SIFMAX = GIFMAX = 63 to 255 > +*** Must support at least 63 distinct interrupt identities > +*** Must implement `seteipnum_le` memory-mapped register > +*** Must implement at least 1 guest interrupt file > +** One, or more AIA APLIC devices to support wired interrupts > +*** EIID and IID fields must be 6 to 8 bits wide matching the number of > + interrrupt identities supported by AIA IMSIC > +** Software interrupts for M-mode, HS-mode and VS-mode supported using > + AIA IMSIC devices > +** MSI virtualization is supported > + > +===== Summary > + > +The <<table_interrutps_and_timer_osa_platforms>> below summarizes the four > +categories of timer and interrupt support allowed in a OS/A platorm. > + > +[#table_interrutps_and_timer_osa_platforms] > +.Interrupts and Timer support in OS/A platforms > +[stripes="none", width="100%", cols="^2s,^1s,^1s,^1s,^1s,^1s,^1s,^1s,^1s,^1s,^1s,^1s,^1s"] > +|=== > +.2+|+++<font size=".8em">OS-A Platform</font>+++ > +3+|+++<font size=".8em">MSIs</font>+++ > +3+|+++<font size=".8em">Wired Interrupts</font>+++ > +3+|+++<font size=".8em">Software Interrupts</font>+++ > +3+|+++<font size=".8em">Timer</font>+++ > +|+++<font size=".6em">M-mode</font>+++ > +|+++<font size=".6em">S-mode</font>+++ > +|+++<font size=".6em">VS-mode</font>+++ > +|+++<font size=".6em">M-mode</font>+++ > +|+++<font size=".6em">S-mode</font>+++ > +|+++<font size=".6em">VS-mode</font>+++ > +|+++<font size=".6em">M-mode</font>+++ > +|+++<font size=".6em">S-mode</font>+++ > +|+++<font size=".6em">VS-mode</font>+++ > +|+++<font size=".6em">M-mode</font>+++ > +|+++<font size=".6em">S-mode</font>+++ > +|+++<font size=".6em">VS-mode</font>+++ > + > +|+++<font size=".8em">Legacy IRQs</font>+++ > +|+++<font size=".6em">NA</font>+++ > +|+++<font size=".6em">NA</font>+++ > +|+++<font size=".6em">NA</font>+++ > +|+++<font size=".6em">PLIC</font>+++ ^<<spec_plic>>^ > +|+++<font size=".6em">PLIC</font>+++ ^<<spec_plic>>^ > +|+++<font size=".6em">PLIC (emulate)</font>+++ ^<<spec_plic>>^ > +|+++<color rgb="#6aa84f"><font size=".6em">MSWI</font></color>+++ ^<<spec_aclint>>^ > +|+++<color rgb="#e06666"><font size=".6em">SBI IPI</font></color>+++ ^<<spec_sbi>>^ > +|+++<color rgb="#e06666"><font size=".6em">SBI IPI</font></color>+++ ^<<spec_sbi>>^ > +|+++<color rgb="#6aa84f"><font size=".6em">MTIMER</font></color>+++ ^<<spec_aclint>>^ > +|+++<color rgb="#e06666"><font size=".6em">SBI Timer</font></color>+++ ^<<spec_sbi>>^ > +|+++<color rgb="#e06666"><font size=".6em">SBI Timer</font></color>+++ ^<<spec_sbi>>^ > + > +|+++<font size=".8em">Only Wired IRQs</font>+++ > +|+++<font size=".6em">NA</font>+++ > +|+++<font size=".6em">NA</font>+++ > +|+++<font size=".6em">NA</font>+++ > +|+++<color rgb="#738dc5"><font size=".6em">APLIC</font></color>+++ ^<<spec_aia>>^ > +|+++<color rgb="#738dc5"><font size=".6em">APLIC</font></color>+++ ^<<spec_aia>>^ > +|+++<color rgb="#738dc5"><font size=".6em">APLIC (emulate)</font></color>+++ ^<<spec_aia>>^ > +|+++<color rgb="#6aa84f"><font size=".6em">MSWI</font></color>+++ ^<<spec_aclint>>^ > +|+++<color rgb="#6aa84f"><font size=".6em">SSWI</font></color>+++ ^<<spec_aclint>>^ > +|+++<color rgb="#e06666"><font size=".6em">SBI IPI</font></color>+++ ^<<spec_sbi>>^ > +|+++<color rgb="#6aa84f"><font size=".6em">MTIMER</font></color>+++ ^<<spec_aclint>>^ > +|+++<color rgb="#e69138"><font size=".6em">Priv Sstc</font></color>+++ ^<<spec_priv_sstc>>^ > +|+++<color rgb="#e69138"><font size=".6em">Priv Sstc</font></color>+++ ^<<spec_priv_sstc>>^ > + > +|+++<font size=".8em">MSIs and Wired IRQs</font>+++ > +|+++<color rgb="#738dc5"><font size=".6em">IMSIC</font></color>+++ ^<<spec_aia>>^ > +|+++<color rgb="#738dc5"><font size=".6em">IMSIC</font></color>+++ ^<<spec_aia>>^ > +|+++<color rgb="#738dc5"><font size=".6em">IMSIC (emulate)</font></color>+++ ^<<spec_aia>>^ > +|+++<color rgb="#738dc5"><font size=".6em">APLIC</font></color>+++ ^<<spec_aia>>^ > +|+++<color rgb="#738dc5"><font size=".6em">APLIC</font></color>+++ ^<<spec_aia>>^ > +|+++<color rgb="#738dc5"><font size=".6em">APLIC (emulate)</font></color>+++ ^<<spec_aia>>^ > +|+++<color rgb="#738dc5"><font size=".6em">IMSIC</font></color>+++ ^<<spec_aia>>^ > +|+++<color rgb="#738dc5"><font size=".6em">IMSIC</font></color>+++ ^<<spec_aia>>^ > +|+++<color rgb="#e06666"><font size=".6em">SBI IPI</font></color>+++ ^<<spec_sbi>>^ > +|+++<color rgb="#6aa84f"><font size=".6em">MTIMER</font></color>+++ ^<<spec_aclint>>^ > +|+++<color rgb="#e69138"><font size=".6em">Priv Sstc</font></color>+++ ^<<spec_priv_sstc>>^ > +|+++<color rgb="#e69138"><font size=".6em">Priv Sstc</font></color>+++ ^<<spec_priv_sstc>>^ > + > +|+++<font size=".8em">MSIs, Virtual MSIs and Wired IRQs</font>+++ > +|+++<color rgb="#738dc5"><font size=".6em">IMSIC</font></color>+++ ^<<spec_aia>>^ > +|+++<color rgb="#738dc5"><font size=".6em">IMSIC</font></color>+++ ^<<spec_aia>>^ > +|+++<color rgb="#738dc5"><font size=".6em">IMSIC</font></color>+++ ^<<spec_aia>>^ > +|+++<color rgb="#738dc5"><font size=".6em">APLIC</font></color>+++ ^<<spec_aia>>^ > +|+++<color rgb="#738dc5"><font size=".6em">APLIC</font></color>+++ ^<<spec_aia>>^ > +|+++<color rgb="#738dc5"><font size=".6em">APLIC (emulate)</font></color>+++ ^<<spec_aia>>^ > +|+++<color rgb="#738dc5"><font size=".6em">IMSIC</font></color>+++ ^<<spec_aia>>^ > +|+++<color rgb="#738dc5"><font size=".6em">IMSIC</font></color>+++ ^<<spec_aia>>^ > +|+++<color rgb="#738dc5"><font size=".6em">IMSIC</font></color>+++ ^<<spec_aia>>^ > +|+++<color rgb="#6aa84f"><font size=".6em">MTIMER</font></color>+++ ^<<spec_aclint>>^ > +|+++<color rgb="#e69138"><font size=".6em">Priv Sstc</font></color>+++ ^<<spec_priv_sstc>>^ > +|+++<color rgb="#e69138"><font size=".6em">Priv Sstc</font></color>+++ ^<<spec_priv_sstc>>^ > +|=== > > ==== System Peripherals > * UART/Serial Console > @@ -359,27 +419,6 @@ of the following: > ** UART 16550 - _REQUIRED_ > ** UART 8250 - _DEPRECATED_ > > -* Clock and Timers > -** Platforms are required to provide an at least 10ns resolution 64-bit counter > -with strictly monotonic updates. > -** The hardware clock that drives the counter is required to operate at a minimum > -frequency of 10MHz. > -** Platforms that use DT for hardware discovery are required to advertise the > -timebase to the operating systems via the `timebase-frequency` property of the > -"/cpus" node > -footnote:[https://elixir.bootlin.com/linux/latest/source/Documentation/devicetree/bindings/riscv/cpus.yaml]. > - > -[sidebar] > --- > -[underline]*_Implementation Note_* > - > -For a counter with 10ns resolution the `timebase-frequency` value would be 100000000 > -(100 MHz) which would also be the minimum possible value for `timebase-frequency`. > -From the software perspective a unit increment of the mtime value would correspond > -to a 10ns interval. However the hardware clock driving the counter could operate at a > -lower frequency, thereby incrementing the mtime value by more than one unit per > -clock tick. > --- > ==== Boot Process > - The base specification defines the interface between the firmware and the > operating system suitable for the RISC-V platforms with rich operating > @@ -449,6 +488,8 @@ base with the additional requirements as below. > The platforms which conform to server extension are required to implement + > > * RV64 support > +* The `time` CSR must be implemented in hardware > +* The Sstc extension <<spec_priv_sstc>> must be implemented > * RISC-V ISA H-extension with following additional requirements: > ** VMID support > ** Upon load/store/AMO exceptions taken into HS-mode, the `htinst` CSR must > @@ -489,7 +530,20 @@ above. > respect to all harts connected to the DM > * Rationale: Debuggers must be able to view memory coherently > > -==== Interrupt Controller > +==== Interrupts and Timer > + > +The OS-A server platform must comply with "MSIs, Virtual MSIs and Wired IRQs" > +category of interrupts and timer support defined in the base with following > +additional requirements: > + > +* The H-extension implemented by each hart must support GEILEN = 3 or more > +* Per-hart AIA IMSIC devices > +** Must implement at least 3 guest interrupt files > +** Must support at least 255 distinct interrupt identities > +** IPRIOLEN = 8 verb missing > +** MIFMAX = SIFMAX = GIFMAX = 255 verb missing [Anup] Okay, will update. Regards, Anup Best regards Heinrich > +* EIID and IID fields of AIA APLIC devices must be at least 8 bits wide > + matching the number of interrrupt identities supported by AIA IMSIC > > ==== Boot and Runtime Requirements > ===== Firmware > @@ -659,12 +713,6 @@ implemented but it can return EFI_UNSUPPORTED. > |=== > > ==== System Peripherals > -===== Clock and Timers > -** Platforms are required to implement the time CSR. > -** Platforms are required to implement the Sstc extension <<spec_priv_sstc>>. > -** Platforms are required to delegate the supervisor timer interrupt to 'S' > -mode. If the 'H' extension is implemented then the platforms are required to > -delegate the virtual supervisor timer interrupt to 'VS' mode. > > ===== Watchdog Timers > Implementation of a two-stage watchdog timer, as defined in the WatchDog Timer > @@ -897,11 +945,6 @@ in the physical RV processor. + > * Logging and/or reporting of errors can be masked. + > * PCIe AER capability is required. + > > -==== Interrupt Controller, Software Interrupt, and Timer Requirements > - * For Timer support, ACLINT MTIMER devices is/are Required > - * For Interrupt Controller and Software Interrupt support, the following one choice is Required > - - Zero, one, or more AIA APLIC devices plus per-hart AIA IMSIC devices > - > // M Platform > == M Platform > > diff --git a/riscv-platform-spec.yml b/riscv-platform-spec.yml > new file mode 100644 > index 0000000..ce00e22 > --- /dev/null > +++ b/riscv-platform-spec.yml > @@ -0,0 +1,8 @@ > +extends: default > +image: > + align: center > +caption: > + align: center > +table: > + caption: > + side: bottom > |
|
|
|
Heinrich Schuchardt
On 8/6/21 8:19 AM, Anup Patel wrote:
We re-write the interrupts and timer section to align theverb missing +** MIFMAX = SIFMAX = GIFMAX = 255verb missing Best regards Heinrich +* EIID and IID fields of AIA APLIC devices must be at least 8 bits wide |
|
|
|
We re-write the interrupts and timer section to align the
interrupts and timer table. We also add more fine grained requirements for AIA. Signed-off-by: Anup Patel <anup.patel@...> --- Makefile | 4 +- riscv-platform-spec.adoc | 323 ++++++++++++++++++++++----------------- riscv-platform-spec.yml | 8 + 3 files changed, 193 insertions(+), 142 deletions(-) create mode 100644 riscv-platform-spec.yml diff --git a/Makefile b/Makefile index 48e8fe9..ff8fbf6 100644 --- a/Makefile +++ b/Makefile @@ -22,8 +22,8 @@ $(PLATFORM_SPEC).md: $(PLATFORM_SPEC).xml $(PLATFORM_SPEC).xml: $(PLATFORM_SPEC).adoc $(ASCIIDOCTOR) -d book -b docbook $< -$(PLATFORM_SPEC).pdf: $(PLATFORM_SPEC).adoc $(IMAGES) - $(ASCIIDOCTOR) -d book -r asciidoctor-pdf -b pdf $< +$(PLATFORM_SPEC).pdf: $(PLATFORM_SPEC).adoc $(IMAGES) $(PLATFORM_SPEC).yml + $(ASCIIDOCTOR) -d book -r asciidoctor-pdf -a pdf-style=$(PLATFORM_SPEC).yml -b pdf $< $(PLATFORM_SPEC).html: $(PLATFORM_SPEC).adoc $(IMAGES) $(ASCIIDOCTOR) -d book -b html $< diff --git a/riscv-platform-spec.adoc b/riscv-platform-spec.adoc index cffb71c..3994ef3 100644 --- a/riscv-platform-spec.adoc +++ b/riscv-platform-spec.adoc @@ -13,6 +13,7 @@ :doctype: book :sectnums: :sectnumlevels: 5 +:xrefstyle: short :toc: macro :toclevels: 5 @@ -235,114 +236,173 @@ each must be 1 The default should allow code that's sensitive to these requirements to be debugged. -==== Interrupt Controller, Software Interrupt, and Timer Requirements -In the following requirements, -https://github.com/riscv/riscv-aia[*AIA*] refers to the Advanced Interrupt Architecture, https://github.com/riscv/riscv-aclint/blob/main/riscv-aclint.adoc[*ACLINT*] -refers to the Advanced *CLINT*. AIA comprises two separate components: `IMSICs` and `APLICs`. -If supported, there is an `IMSIC` device associated with each hart. -If supported, *APLIC* devices are global to all harts, and there may be one or -multiple in a system. *ACLINT* comprises three separate components: `MTIMER` for -Timer support, and `MSWI` and `SSWI` for Machine-level and Supervisor-level -Software Interrupt (IPI) support. + -https://github.com/riscv/riscv-plic-spec/blob/master/riscv-plic.adoc[*PLIC*] -refers to the legacy Platform-Level Interrupt Controller that provides -facilities to route external interrupts to a hart context with a given privilege -mode. The number of non-local interrupt sources supported by PLIC and how does -each of them connect to the hart context is PLIC core implementation-specific. + -*CLINT* is a legacy Core-Local Interrupt Controller that is a compatible subset of -ACLINT which provides facilities to trigger Software (IPI) and Timer interrupts to -hart. +==== Interrupts and Timer + +===== Timer support + +* One or more ACLINT MTIMER devices are required for the OS-A platform +* All MTIME counters must be 64-bit counters with a 10ns resolution (i.e. + an increment by 1 represents 10 ns). +* The MTIME update frequency (i.e. hardware clock) must be between 10 MHz + and 100 MHz, and updates must be strictly monotonic. +* Platforms must provide a way to discover MTIME resolution using appropriate + hardware discovery mechanism: +** Platforms that use DT for hardware discovery are required to advertise the + timebase to the operating systems via the `timebase-frequency` property of + the "/cpus" node +footnote:[https://elixir.bootlin.com/linux/latest/source/Documentation/devicetree/bindings/riscv/cpus.yaml]. + +[sidebar] +-- +[underline]*_Implementation Note_* -.The following table summarizes what features are supported for four classes of OS/A platforms. -[width="100%",cols="^,^,^,^,^,^,^,^,^,^,^,^,^"] -|======= -.2+|*OS-A Platform* 3+|*MSIs* 3+|*Wired Interrupts* 3+|*IPIs* 3+|*Timer* -|M-level|S-level|VS-level|M-level|S-level|VS-level|M-level|S-level|VS-level|M-level|S-level|VS-level -|Existing|NA|NA|NA|PLIC|PLIC|PLIC + -_(Trap and emulate)_|MSWI + -https://github.com/riscv/riscv-aclint/blob/main/riscv-aclint.adoc[`*ACLINT*`]| -SBI IPI + -https://github.com/riscv/riscv-sbi-doc[`*SBI*`]|SBI IPI + -https://github.com/riscv/riscv-sbi-doc[`*SBI*`]|MTIMER + -https://github.com/riscv/riscv-aclint/blob/main/riscv-aclint.adoc[`*ACLINT*`]|SBI Timer + -https://github.com/riscv/riscv-sbi-doc[`*SBI*`]|SBI Timer + -https://github.com/riscv/riscv-sbi-doc[`*SBI*`] -|Only Wired IRQs|NA|NA|NA|APLIC M-level + -https://github.com/riscv/riscv-aia[`*AIA*`]|APLIC S-level + -https://github.com/riscv/riscv-aia[`*AIA*`]|APLIC S-level + -_(Trap and emulate)_ + -https://github.com/riscv/riscv-aia[`*AIA*`]|MSWI + -https://github.com/riscv/riscv-aclint/blob/main/riscv-aclint.adoc[`*ACLINT*`]| -SSWI + -https://github.com/riscv/riscv-aclint/blob/main/riscv-aclint.adoc[`*ACLINT*`]| -SBI IPI + -https://github.com/riscv/riscv-sbi-doc[`*SBI*`]|MTIMER + -https://github.com/riscv/riscv-aclint/blob/main/riscv-aclint.adoc[`*ACLINT*`]| -Priv Sstc + -https://github.com/riscv/riscv-isa-manual/releases[`*Priv Spec*`]| -Priv Sstc + -https://github.com/riscv/riscv-isa-manual/releases[`*Priv Spec*`] -|MSIs and Wired IRQs|IMSIC M-level + -https://github.com/riscv/riscv-aia[`*AIA*`]|IMSIC S-level + -https://github.com/riscv/riscv-aia[`*AIA*`]|APLIC S-level + -_(Trap and emulate)_ + -https://github.com/riscv/riscv-aia[`*AIA*`]|APLIC M-level + -https://github.com/riscv/riscv-aia[`*AIA*`]|APLIC S-level + -https://github.com/riscv/riscv-aia[`*AIA*`]|APLIC S-level + -_(Trap and emulate)_ + -https://github.com/riscv/riscv-aia[`*AIA*`]|IMSIC M-level + -https://github.com/riscv/riscv-aia[`*AIA*`]|IMSIC S-level + -https://github.com/riscv/riscv-aia[`*AIA*`]|SBI IPI + -https://github.com/riscv/riscv-sbi-doc[`*SBI*`]|MTIMER + -https://github.com/riscv/riscv-aclint/blob/main/riscv-aclint.adoc[`*ACLINT*`]| -Priv Sstc + -https://github.com/riscv/riscv-isa-manual/releases[`*Priv Spec*`]| -Priv Sstc + -https://github.com/riscv/riscv-isa-manual/releases[`*Priv Spec*`] -|MSIs, Virtual MSIs and Wired IRQs|IMSIC M-level + -https://github.com/riscv/riscv-aia[`*AIA*`]|IMSIC S-level + -https://github.com/riscv/riscv-aia[`*AIA*`]|APLIC VS-level + -https://github.com/riscv/riscv-aia[`*AIA*`]|APLIC M-level + -https://github.com/riscv/riscv-aia[`*AIA*`]|APLIC S-level + -https://github.com/riscv/riscv-aia[`*AIA*`]|APLIC S-level + -_(Trap and emulate)_ + -https://github.com/riscv/riscv-aia[`*AIA*`]|IMSIC M-level + -https://github.com/riscv/riscv-aia[`*AIA*`]|IMSIC S-level + -https://github.com/riscv/riscv-aia[`*AIA*`]|IMSIC VS-level + -https://github.com/riscv/riscv-aia[`*AIA*`]|MTIMER + -https://github.com/riscv/riscv-aclint/blob/main/riscv-aclint.adoc[`*ACLINT*`]| -Priv Sstc + -https://github.com/riscv/riscv-isa-manual/releases[`*Priv Spec*`]| -Priv Sstc + -https://github.com/riscv/riscv-isa-manual/releases[`*Priv Spec*`] -|======= - -* For Timer support, one or more ACLINT MTIMER devices are Required for OS-A platform. -One MTIMER may be used for all harts, or multiple MTIMERs may be used with -multiple topological groups of harts. The base address of MTIMER memory map registers -is platform implementation-specific, however, the format of MTIMER operation parameters -(`mtime` and `mtimecmp` registers) must be compliant with -https://github.com/riscv/riscv-aclint/blob/main/riscv-aclint.adoc#21-register-map[ACLINT MTIMER Register Map] - -* For Interrupt Controller and Software Interrupt support, one of the following three -choices below is Required - ** PLIC plus one or more ACLINT MSWI devices - DEPRECATED - *** One MSWI may be used for all harts, or multiple MSWIs may be used with -multiple topological groups of harts - *** Only wired interrupts and M-mode IPIs are supported - *** Virtualization is not supported - *** This compatibly supports legacy PLIC + CLINT designs - ** One or more AIA APLIC devices plus one or more pairs of ACLINT MSWI and ACLINT SSWI devices - *** One MSWI/SSWI pair may be used for all harts, or multiple MSWI/SSWI -pairs may be used with multiple topological groups of harts - *** Only wired interrupts are supported - *** Both M-mode and S-mode IPIs are supported - *** Virtualization is not supported - ** Zero, one, or more AIA APLIC devices plus per-hart AIA IMSIC devices - *** Both wired and MSI external interrupts are supported - *** Both M-mode and S-mode IPIs are supported via IMSICs - *** Virtualization is supported - *** Zero APLICs if there are no wired interrupts and only MSIs +For example, if `timebase-frequency` is 100 MHz and the MTIME update +frequency (i.e. hardware clock) is 25 MHz then the MTIME counter would +increment by 4 upon every hardware clock tick. +-- + +===== Interrupts support + +The OS/A platorms can comply with one of the following four categories for +interrupts support: + +* *Legacy IRQs - DEPRECATED* +** One or more PLIC devices to support wired interrupts +** One or more ACLINT MSWI devices to support M-mode software interrupts +** Software interrupts for S-mode and VS-mode supported using the + SBI IPI extension +** Compatibile with legacy platforms having PLIC plus CLINT devices +** MSI external interrupts not supported +** MSI virtualization is not supported +* *Only Wired IRQs* +** One or more AIA APLIC devices to support wired interrupts +** One or more ACLINT MSWI devices to support M-mode software interrupts +** One or more ACLINT SSWI devices to support HS-mode software interrupts +** Software interrupts for VS-mode supported using the SBI IPI extension +** MSI external interrupts not supported +** MSI virtualization is not supported +* *MSIs and Wired IRQs* +** AIA local interrupt CSRs must be supported by each hart +*** `mvien` and `mvip` CSRs must be implemented +*** `hvien` and `hvip` CSRs must be implemented if H-extension is implemented +*** `siselect` CSR must support holding a 9-bit value +*** `vsiselect` CSR must support holding a 9-bit value if H-extension + is implemented +** Per-hart AIA IMSIC devices must support MSIs for M-mode and HS-mode +*** IPRIOLEN = 6 to 8 +*** MIFMAX = SIFMAX = 63 to 255 +*** Must support at least 63 distinct interrupt identities +*** Must implement `seteipnum_le` memory-mapped register +** One, or more AIA APLIC devices to support wired interrupts +*** EIID and IID fields must be 6 to 8 bits wide matching the number of + interrrupt identities supported by AIA IMSIC +** Software interrupts for M-mode and HS-mode supported using AIA IMSIC devices +** Software interrupts for VS-mode supported using the SBI IPI extension +** MSI virtualization is not supported +* *MSIs, Virtual MSIs, and Wired IRQs* +** The H-extension must be implemeted by each hart to support virtual MSIs +*** GEILEN must be 1 or more +** AIA local interrupt CSRs must be supported by each hart +*** `mvien` and `mvip` CSRs must be implemented +*** `hvien` and `hvip` CSRs must be implemented +*** `siselect` CSR must support holding a 9-bit value +*** `vsiselect` CSR must support holding a 9-bit value +** Per-hart AIA IMSIC devices to support MSIs for M-mode, HS-mode and VS-mode +*** IPRIOLEN = 6 to 8 +*** MIFMAX = SIFMAX = GIFMAX = 63 to 255 +*** Must support at least 63 distinct interrupt identities +*** Must implement `seteipnum_le` memory-mapped register +*** Must implement at least 1 guest interrupt file +** One, or more AIA APLIC devices to support wired interrupts +*** EIID and IID fields must be 6 to 8 bits wide matching the number of + interrrupt identities supported by AIA IMSIC +** Software interrupts for M-mode, HS-mode and VS-mode supported using + AIA IMSIC devices +** MSI virtualization is supported + +===== Summary + +The <<table_interrutps_and_timer_osa_platforms>> below summarizes the four +categories of timer and interrupt support allowed in a OS/A platorm. + +[#table_interrutps_and_timer_osa_platforms] +.Interrupts and Timer support in OS/A platforms +[stripes="none", width="100%", cols="^2s,^1s,^1s,^1s,^1s,^1s,^1s,^1s,^1s,^1s,^1s,^1s,^1s"] +|=== +.2+|+++<font size=".8em">OS-A Platform</font>+++ +3+|+++<font size=".8em">MSIs</font>+++ +3+|+++<font size=".8em">Wired Interrupts</font>+++ +3+|+++<font size=".8em">Software Interrupts</font>+++ +3+|+++<font size=".8em">Timer</font>+++ +|+++<font size=".6em">M-mode</font>+++ +|+++<font size=".6em">S-mode</font>+++ +|+++<font size=".6em">VS-mode</font>+++ +|+++<font size=".6em">M-mode</font>+++ +|+++<font size=".6em">S-mode</font>+++ +|+++<font size=".6em">VS-mode</font>+++ +|+++<font size=".6em">M-mode</font>+++ +|+++<font size=".6em">S-mode</font>+++ +|+++<font size=".6em">VS-mode</font>+++ +|+++<font size=".6em">M-mode</font>+++ +|+++<font size=".6em">S-mode</font>+++ +|+++<font size=".6em">VS-mode</font>+++ + +|+++<font size=".8em">Legacy IRQs</font>+++ +|+++<font size=".6em">NA</font>+++ +|+++<font size=".6em">NA</font>+++ +|+++<font size=".6em">NA</font>+++ +|+++<font size=".6em">PLIC</font>+++ ^<<spec_plic>>^ +|+++<font size=".6em">PLIC</font>+++ ^<<spec_plic>>^ +|+++<font size=".6em">PLIC (emulate)</font>+++ ^<<spec_plic>>^ +|+++<color rgb="#6aa84f"><font size=".6em">MSWI</font></color>+++ ^<<spec_aclint>>^ +|+++<color rgb="#e06666"><font size=".6em">SBI IPI</font></color>+++ ^<<spec_sbi>>^ +|+++<color rgb="#e06666"><font size=".6em">SBI IPI</font></color>+++ ^<<spec_sbi>>^ +|+++<color rgb="#6aa84f"><font size=".6em">MTIMER</font></color>+++ ^<<spec_aclint>>^ +|+++<color rgb="#e06666"><font size=".6em">SBI Timer</font></color>+++ ^<<spec_sbi>>^ +|+++<color rgb="#e06666"><font size=".6em">SBI Timer</font></color>+++ ^<<spec_sbi>>^ + +|+++<font size=".8em">Only Wired IRQs</font>+++ +|+++<font size=".6em">NA</font>+++ +|+++<font size=".6em">NA</font>+++ +|+++<font size=".6em">NA</font>+++ +|+++<color rgb="#738dc5"><font size=".6em">APLIC</font></color>+++ ^<<spec_aia>>^ +|+++<color rgb="#738dc5"><font size=".6em">APLIC</font></color>+++ ^<<spec_aia>>^ +|+++<color rgb="#738dc5"><font size=".6em">APLIC (emulate)</font></color>+++ ^<<spec_aia>>^ +|+++<color rgb="#6aa84f"><font size=".6em">MSWI</font></color>+++ ^<<spec_aclint>>^ +|+++<color rgb="#6aa84f"><font size=".6em">SSWI</font></color>+++ ^<<spec_aclint>>^ +|+++<color rgb="#e06666"><font size=".6em">SBI IPI</font></color>+++ ^<<spec_sbi>>^ +|+++<color rgb="#6aa84f"><font size=".6em">MTIMER</font></color>+++ ^<<spec_aclint>>^ +|+++<color rgb="#e69138"><font size=".6em">Priv Sstc</font></color>+++ ^<<spec_priv_sstc>>^ +|+++<color rgb="#e69138"><font size=".6em">Priv Sstc</font></color>+++ ^<<spec_priv_sstc>>^ + +|+++<font size=".8em">MSIs and Wired IRQs</font>+++ +|+++<color rgb="#738dc5"><font size=".6em">IMSIC</font></color>+++ ^<<spec_aia>>^ +|+++<color rgb="#738dc5"><font size=".6em">IMSIC</font></color>+++ ^<<spec_aia>>^ +|+++<color rgb="#738dc5"><font size=".6em">IMSIC (emulate)</font></color>+++ ^<<spec_aia>>^ +|+++<color rgb="#738dc5"><font size=".6em">APLIC</font></color>+++ ^<<spec_aia>>^ +|+++<color rgb="#738dc5"><font size=".6em">APLIC</font></color>+++ ^<<spec_aia>>^ +|+++<color rgb="#738dc5"><font size=".6em">APLIC (emulate)</font></color>+++ ^<<spec_aia>>^ +|+++<color rgb="#738dc5"><font size=".6em">IMSIC</font></color>+++ ^<<spec_aia>>^ +|+++<color rgb="#738dc5"><font size=".6em">IMSIC</font></color>+++ ^<<spec_aia>>^ +|+++<color rgb="#e06666"><font size=".6em">SBI IPI</font></color>+++ ^<<spec_sbi>>^ +|+++<color rgb="#6aa84f"><font size=".6em">MTIMER</font></color>+++ ^<<spec_aclint>>^ +|+++<color rgb="#e69138"><font size=".6em">Priv Sstc</font></color>+++ ^<<spec_priv_sstc>>^ +|+++<color rgb="#e69138"><font size=".6em">Priv Sstc</font></color>+++ ^<<spec_priv_sstc>>^ + +|+++<font size=".8em">MSIs, Virtual MSIs and Wired IRQs</font>+++ +|+++<color rgb="#738dc5"><font size=".6em">IMSIC</font></color>+++ ^<<spec_aia>>^ +|+++<color rgb="#738dc5"><font size=".6em">IMSIC</font></color>+++ ^<<spec_aia>>^ +|+++<color rgb="#738dc5"><font size=".6em">IMSIC</font></color>+++ ^<<spec_aia>>^ +|+++<color rgb="#738dc5"><font size=".6em">APLIC</font></color>+++ ^<<spec_aia>>^ +|+++<color rgb="#738dc5"><font size=".6em">APLIC</font></color>+++ ^<<spec_aia>>^ +|+++<color rgb="#738dc5"><font size=".6em">APLIC (emulate)</font></color>+++ ^<<spec_aia>>^ +|+++<color rgb="#738dc5"><font size=".6em">IMSIC</font></color>+++ ^<<spec_aia>>^ +|+++<color rgb="#738dc5"><font size=".6em">IMSIC</font></color>+++ ^<<spec_aia>>^ +|+++<color rgb="#738dc5"><font size=".6em">IMSIC</font></color>+++ ^<<spec_aia>>^ +|+++<color rgb="#6aa84f"><font size=".6em">MTIMER</font></color>+++ ^<<spec_aclint>>^ +|+++<color rgb="#e69138"><font size=".6em">Priv Sstc</font></color>+++ ^<<spec_priv_sstc>>^ +|+++<color rgb="#e69138"><font size=".6em">Priv Sstc</font></color>+++ ^<<spec_priv_sstc>>^ +|=== ==== System Peripherals * UART/Serial Console @@ -359,27 +419,6 @@ of the following: ** UART 16550 - _REQUIRED_ ** UART 8250 - _DEPRECATED_ -* Clock and Timers -** Platforms are required to provide an at least 10ns resolution 64-bit counter -with strictly monotonic updates. -** The hardware clock that drives the counter is required to operate at a minimum -frequency of 10MHz. -** Platforms that use DT for hardware discovery are required to advertise the -timebase to the operating systems via the `timebase-frequency` property of the -"/cpus" node -footnote:[https://elixir.bootlin.com/linux/latest/source/Documentation/devicetree/bindings/riscv/cpus.yaml]. - -[sidebar] --- -[underline]*_Implementation Note_* - -For a counter with 10ns resolution the `timebase-frequency` value would be 100000000 -(100 MHz) which would also be the minimum possible value for `timebase-frequency`. -From the software perspective a unit increment of the mtime value would correspond -to a 10ns interval. However the hardware clock driving the counter could operate at a -lower frequency, thereby incrementing the mtime value by more than one unit per -clock tick. --- ==== Boot Process - The base specification defines the interface between the firmware and the operating system suitable for the RISC-V platforms with rich operating @@ -449,6 +488,8 @@ base with the additional requirements as below. The platforms which conform to server extension are required to implement + * RV64 support +* The `time` CSR must be implemented in hardware +* The Sstc extension <<spec_priv_sstc>> must be implemented * RISC-V ISA H-extension with following additional requirements: ** VMID support ** Upon load/store/AMO exceptions taken into HS-mode, the `htinst` CSR must @@ -489,7 +530,20 @@ above. respect to all harts connected to the DM * Rationale: Debuggers must be able to view memory coherently -==== Interrupt Controller +==== Interrupts and Timer + +The OS-A server platform must comply with "MSIs, Virtual MSIs and Wired IRQs" +category of interrupts and timer support defined in the base with following +additional requirements: + +* The H-extension implemented by each hart must support GEILEN = 3 or more +* Per-hart AIA IMSIC devices +** Must implement at least 3 guest interrupt files +** Must support at least 255 distinct interrupt identities +** IPRIOLEN = 8 +** MIFMAX = SIFMAX = GIFMAX = 255 +* EIID and IID fields of AIA APLIC devices must be at least 8 bits wide + matching the number of interrrupt identities supported by AIA IMSIC ==== Boot and Runtime Requirements ===== Firmware @@ -659,12 +713,6 @@ implemented but it can return EFI_UNSUPPORTED. |=== ==== System Peripherals -===== Clock and Timers -** Platforms are required to implement the time CSR. -** Platforms are required to implement the Sstc extension <<spec_priv_sstc>>. -** Platforms are required to delegate the supervisor timer interrupt to 'S' -mode. If the 'H' extension is implemented then the platforms are required to -delegate the virtual supervisor timer interrupt to 'VS' mode. ===== Watchdog Timers Implementation of a two-stage watchdog timer, as defined in the WatchDog Timer @@ -897,11 +945,6 @@ in the physical RV processor. + * Logging and/or reporting of errors can be masked. + * PCIe AER capability is required. + -==== Interrupt Controller, Software Interrupt, and Timer Requirements - * For Timer support, ACLINT MTIMER devices is/are Required - * For Interrupt Controller and Software Interrupt support, the following one choice is Required - - Zero, one, or more AIA APLIC devices plus per-hart AIA IMSIC devices - // M Platform == M Platform diff --git a/riscv-platform-spec.yml b/riscv-platform-spec.yml new file mode 100644 index 0000000..ce00e22 --- /dev/null +++ b/riscv-platform-spec.yml @@ -0,0 +1,8 @@ +extends: default +image: + align: center +caption: + align: center +table: + caption: + side: bottom -- 2.25.1 |
|
|