Why aren't my two Cortex-A9 cores cache coherent?

Posted on 2026-08-22

Contents

One of my favourite things to do when I have some down time is to grab a random devkit from my pile of random devkits, and try and write some code that will run on it. I generally try and avoid existing SDKs, or installing new tools. I just want to see how little I can write and have something run on the board and have an observable output. If that goes OK, I might go overboard and start writing some little drivers for a few peripherals here and there. If it really gets out of control I might end up having written a whole new Operating System, with VGA output and SD Card support. But usually I'm OK with Hello, World on a UART or a blinking LED.

I had some time off over the past few weeks and, looking for a change from playing with ARM7, I picked up my Terasic DE0-Nano-SOC. I've had this board for about 7 years but I hadn't really done anything with it. It's similar to the DE10 Nano board that is used by the MiSTER retro emulation project, but not compatible.

The Hardware

The Terasic DE0-Nano-SOC is a small devkit released in around 2015 from what I can tell. It has:

  • An Altera Cyclone-V System on Chip
  • 1 GiB DDR3 SDRAM
  • An RJ45 Ethernet port
  • A 5V power input (barrel-jack)
  • A mini-USB programming port
  • A micro-USB port for an on-board UART to USB Serial convertor
  • An SD Card slot
  • Arduino Uno style headers

Sadly what it doesn't have is an active cooler, because when running it gets really hot. Like, too hot to touch. I've wired up a spare 80mm fan which seems to do more than enough to keep it cool.

The Altera Cyclone-V SoC is a combination of a standard Arm SoC (the Hard Processor System) and an Altera FPGA (with 40K logic elements). In particular, I was interested in the two Arm Cortex-A9 processors.

The SoC has a built-in Boot ROM that can load a preloader from some special sectors on an SD Card. That preloader will run from On-Chip RAM (OCRAM) and initialise the external DDR3 SDRAM, before loading a full copy of U-Boot from the SD Card into SDRAM and then executing it from there. I didn't want to muck around with all that, so I just downloaded the disk image from https://soc.terasic.com and wrote it to a spare card. It all seemed to boot OK.

By default it boots into a fairly old Linux kernel (with a root partition stored on the SD Card), but I deleted the kernel causing it to stop at the U-Boot prompt. The on-board USB to Serial adapter means I can just hook up my PC to the micro-USB port, and use minicom on /dev/tty.usbserial-<something> and interact with the board.

The preloader prints this:

```
U-Boot SPL 2013.01.01 (Dec 29 2014 - 15:29:15)
BOARD : Terasic DE0_Nano_SoC Version-A Board

############################= =
##=-=#######################= -- =
- =#######################= ####- =###### =
- ###############- -=##=##= ####- ######## =

= =##- -### ##= -####### ###### ####- #####- - =

## ### ## -##= -=--=###= ####==# ####- #### =

- ###- ##- -### =####= =====###= ####- #### #### =
- ### =##### #####= ####-=###= -=###- ####- ##### =
- -#= - =## #####= #### -###= #=-=#### ####- ########-
# =##= ## #####= -########= ####### ####- =###### =
###==######====####==#######= - --- -- ---- =
############################= =

BOARD : Terasic DE0_Nano_SoC Version-A Board
CLOCK: EOSC1 clock 25000 KHz
CLOCK: EOSC2 clock 25000 KHz
CLOCK: F2S_SDR_REF clock 0 KHz
CLOCK: F2S_PER_REF clock 0 KHz
CLOCK: MPU clock 925 MHz
CLOCK: DDR clock 400 MHz
CLOCK: UART clock 100000 KHz
CLOCK: MMC clock 50000 KHz
CLOCK: QSPI clock 3613 KHz
SDRAM: Initializing MMR registers
SDRAM: Calibrating PHY
SEQ.C: Preparing to start memory calibration
SEQ.C: CALIBRATION PASSED
SDRAM: 1024 MiB
ALTERA DWMMC: 0
```
It's just a small copy of U-Boot, linked to run from the small amount of On-Chip RAM (OCRAM).

The full U-Boot then prints this:

```
U-Boot 2013.01.01 (Dec 30 2014 - 12:07:34)
CPU : Altera SOCFPGA Platform
BOARD : Terasic DE0_Nano_SoC Version-A Board

############################= =
##=-=#######################= -- =
- =#######################= ####- =###### =
- ###############- -=##=##= ####- ######## =

= =##- -### ##= -####### ###### ####- #####- - =

## ### ## -##= -=--=###= ####==# ####- #### =

- ###- ##- -### =####= =====###= ####- #### #### =
- ### =##### #####= ####-=###= -=###- ####- ##### =
- -#= - =## #####= #### -###= #=-=#### ####- ########-
# =##= ## #####= -########= ####### ####- =###### =
###==######====####==#######= - --- -- ---- =
############################= =

BOARD : Terasic DE0_Nano_SoC Version-A Board
I2C: ready
DRAM: 1 GiB
MMC: ALTERA DWMMC: 0
In: serial
Out: serial
Err: serial
Skipped ethaddr assignment due to invalid EMAC address in EEPROM
Net: mii0
Warning: failed to set MAC address
Hit any key to stop autoboot: 0
SOCFPGA_CYCLONE5 #
```

Loading Code

I've written enough AArch32 Rust examples by now that it was relatively easy to write another one.

  • Start with an empty project
  • Bring in aarch32-rt
  • Write a memory.xlinker script fragment that says where memory is
  • Write a very basic 16550 UART driver, and point it at the base address of UART0, assuming U-Boot will have left it enabled and configured at a suitable baud rate

The linker script fragment looked like:

MEMORY { RAM : ORIGIN = 0x00100000, LENGTH = 1M } REGION_ALIAS("VECTORS", RAM); REGION_ALIAS("CODE", RAM); REGION_ALIAS("DATA", RAM); REGION_ALIAS("STACKS", RAM); PROVIDE(_vector_start = ORIGIN(VECTORS)); PROVIDE(_hyp_stack_size = 16K); PROVIDE(_und_stack_size = 16K); PROVIDE(_svc_stack_size = 16K); PROVIDE(_abt_stack_size = 16K); PROVIDE(_irq_stack_size = 64); PROVIDE(_fiq_stack_size = 64); PROVIDE(_sys_stack_size = 16K);
We've got a lot of memory to play with, but 1 MiB is more that enough for what we need. The important thing is that the start address is not 0x0. Instead, I chose to set the start address to 0x0010_0000 to skip the first 1 MiB. This was to avoid the Boot ROM (which can be mapped in or out at address 0x0), and to avoid whatever RAM U-Boot was using.

Our crappy UART driver is as simple as:

/// This is the same console that U-Boot uses on the DE0-Nano-SOC pub static CONSOLE: Console = Console::new(); /// Represents our standard-output console (on UART0) pub struct Console { _inner: (), } impl Console { const UART0_BASE_THR: *mut u32 = 0xFFC0_2000 as *mut u32; const UART0_BASE_LSR: *mut u32 = 0xFFC0_2014 as *mut u32; const LSR_TX_EMPTY: u32 = 1 << 6; const fn new() -> Console { Console { _inner: () } } /// Wait while the UART is busy fn waitbusy(&self) { loop { let lsr = unsafe { Self::UART0_BASE_LSR.read_volatile() }; if (lsr & Self::LSR_TX_EMPTY) != 0 { break; } } } /// Put a byte into the UART fn putc(&self, byte: u8) { // Safety: This is our UART and buffer overflows are not UB unsafe { Self::UART0_BASE_THR.write_volatile(byte as u32); } } } impl core::fmt::Write for &Console { fn write_str(&mut self, s: &str) -> core::fmt::Result { for b in s.as_bytes() { self.waitbusy(); if cfg!(feature = "console-crlf") { if *b == b'\n' { self.putc(b'\r'); self.waitbusy(); } } self.putc(*b); } Ok(()) } }
This is basically lifted from an earlier project I did on the Pandaboard (did I mention that this is not my first Arm dev kit?), but with the base address changed. There are a bunch of library crates you could pull in which implement a much better driver, but I'm happy copy-pasting these few lines because it's easier to hack on it when it's in the tree with the rest of the code.

The main function is a simple:

```

![no_std]

![no_main]

use core::fmt::Write;
use hello_de0_nano_soc::CONSOLE;

[aarch32_rt::entry]

fn main() -> ! {
_ = writeln!(&CONSOLE, "Hello, this is a DE0-Nano-SOC!");
panic!("I am a sample panic!");
}
`` Now, to get the code onto the board we need a file format that U-Boot likes. I don't think you can just give it ELF files (shame, I wrote a lovely bare-metal ELF parser so I know it's not that hard), but you can give it Motorola S-Record files. I know, how quaint. Luckily, LLVM's binutils can do that, which I like to drive using thecargo-binutilsplugin forcargo`:

cargo objcopy --release -- -O srecAnnoyingly this overwrites the ELF file with the hex file - answers to the usual address if you know a way to get the objcopy sub-command from cargo-binutils to not do that. But it's fine as I don't need the ELF anyway.

We get U-Boot to load the file by running loads, and sending the file as ASCII through the serial terminal on my Mac. For reasons I don't fully understand I chose to use minicom, and once I'd worked out that "Esc" and then "S" opened the send menu, and "double tap space" enters a directory in the file browser inside minicom, we were off.

The Memory Management Unit

The program runs without issue, but we don't have:

  • The second core running
  • The L1 Instruction Cache enabled
  • The L1 Data Cache enabled
  • The L2 Cache enabled
  • The MMU enabled

The MMU is the really important one because without it, the processor treats all memory as strongly-ordered, and unaligned loads or atomic accesses don't work with that kind of memory. I think technically it's Undefined Behaviour to execute Rust code when you're in that state but whatever, let's just get the MMU up and running.

To do this, we need an array of 4,096 Level 1 page table entries, each 32-bits in length and each representing a 1 MiB portion of the virtual address space. We can use a Rust const fn to generate that at compile time.

```
/// Holds an L1 page table with appropriate alignment
///
/// You should create a static variable of this type, to represent your page table.

[repr(C)]

[derive(Debug)]

pub struct L1Table {
/// Our mutable list of MMU table entries
///
/// This table is read by the hardware.
pub entries: core::cell::UnsafeCell<[L1Section; NUM_L1_PAGE_TABLE_ENTRIES]>,
}
unsafe impl Sync for L1Table {}
/// Our MMU page table

[unsafe(no_mangle)]

[unsafe(link_section = ".pagetable")]

pub static MMU_L1_PAGE_TABLE: L1Table = make_mmu_table();
const DDR_ATTRS: SectionAttributes = SectionAttributes {
non_global: false,
p_bit: false,
shareable: true,
access: AccessPermissions::FullAccess,
memory_attrs: MemoryRegionAttributes::CacheableMemory {
inner: CachePolicy::WriteBackWriteAlloc,
outer: CachePolicy::NonCacheable,
}
.as_raw(),
domain: u4::new(0b0),
execute_never: false,
};
const DEVICE_ATTRS: SectionAttributes = SectionAttributes {
non_global: false,
p_bit: false,
shareable: true,
access: AccessPermissions::FullAccess,
memory_attrs: MemoryRegionAttributes::ShareableDevice.as_raw(),
domain: u4::new(0b0),
execute_never: false,
};
/// The number of bytes in 1 MiB
const ONE_MB: u32 = 1024 * 1024;
const fn make_mmu_table() -> L1Table {
let mut temp: [L1Section; NUM_L1_PAGE_TABLE_ENTRIES] =
[L1Section::ZERO; NUM_L1_PAGE_TABLE_ENTRIES];
let mut page = 0;
// Map 1024 MiB of DDR SDRAM @ 0x0000_0000
while page < 1024 {
let section = L1Section::new_with_addr_and_attrs(0x0000_0000 + (page * ONE_MB), DDR_ATTRS);
temp[0x000 + (page as usize)] = section;
page += 1;
}
// Map 256 MiB of system / MPCore peripherals @ 0xF000_0000
page = 0;
while page < 256 {
let section =
L1Section::new_with_addr_and_attrs(0xF000_0000 + (page * ONE_MB), DEVICE_ATTRS);
temp[0xF00 + (page as usize)] = section;
page += 1;
}
L1Table {
entries: core::cell::UnsafeCell::new(temp),
}
}
`` Initially I used theaarch32_cpu::mmu::L1Table` type, but that is marked as requiring alignment to a 1 MiB boundary. That's fine, except the size of an object must be a multiple of its alignment, so Rust padded the page table out from 16 KiB to 1 MiB. Which is a problem when I'm loading it over a UART at 115,200 baud. So instead I made my own type with no alignment requirements, and put it into a special section to ensure it was aligned appropriately. The Motorola S-Record format has no problem leaving out the gaps, so the load didn't take too long (about 5 seconds or so).

The MMU mapping is very simple - a flat 1:1 mapping from Virtual Address to Physical Address, with the bottom 1 GiB being Inner Cacheable and the top 256 MiB being Device Memory.

Wait, what?

Kinds of Memory

Arm processors understand there are different kinds of memory, and they do this for performance.

Some memory is the kind where if the code writes a 32-bit value to a specific address, the hardware needs to actually do that write, to that address, exactly once, and not before or after any other write that might occur that address (or similar addresses). This is important when the address in question is the UART Transmit FIFO register, for example. Arm call this Device Memory, and it is non-cacheable and strongly-ordered.

If all RAM was treated like this, it would kill your performance. Your RAM is much much much much slower than your processor, and so we need caches (several levels of caches in fact)Â to keep the processor fed with instructions and data as much as possible. This has been true on desktop PCs since the early 1990s (the Intel 486 has an on-die 8 KiB Level 1 cache, for example), and it's been true for Arm processors since ARM3 came out at around the same time.

Our Arm Cortex-A9 processor has two interfaces to memory - one for instructions and one for data (a so-called Modified Harvard Architecture design) - and so it has two Level 1 caches built into the processor. They are often called the I Cache and the D cache for short. We want the processor to use them so we tell the MMU that most of our memory space is Normal Memory. That allows it to cache reads and writes, buffer writes (so they may appear at the caches out-of-order or be coalesced into a single larger write), and generally do things that make CPU go vroom but that we only get away with when address space is backed by RAM and not peripherals pretending to be RAM.

As a side note, when it comes to being cacheable, I see the terms Inner and Outer a lot. I believe Inner is "other processors in the same cluster" and Outer is "things outside that cluster, like other processor clusters, or peripherals that are doing DMA".

On an Armv7-A architecture processor you turn on the L1 I Cache by setting the SCTLR.I bit, and you turn on the L1 D Cache by setting the SCTLR.C bit. There are other bits too, like the SCTLR.M bit to enable the MMU, or the SCTLR.Z bit to turn on branch prediction. I think technically you are supposed to invalidate the cache contents before you enable the caches too, in case your processor didn't do that automatically when it came out of reset.

With my previous examples that ran on QEMU, this was entirely sufficient. However, I have a problem:

  • There's a second processor core I want to enable (in SMP mode),
  • and the two processors have different L1 caches,
  • and they apparently have a mechanism that allows them to 'see' into each others caches?

They also share an L2 cache, which is controlled by a peripheral that is separate from that two Cortex-A9 cores. It's a piece of IP Altera bought from Arm called the Corelink L2C-310 L2 Cache Controller. Bringing up the L2C-310 is a right pain, and I don't actually think I need it enabled to get SMP to work, but I've written a Rust driver for it now.

The actual issue

Using some changes I put in the latest (unreleased) version of aarch32-rt, we can use the library to bring up secondary processor cores, after the primary code has finished initialising all the global variables and generally decided that it's safe for those secondary cores to start running. The aarch32-rt library knows how to read a special register that identifies each processor core with a number, and how to ensure each processor core gets a unique allocation for each of its seven (7!!) stacks. All we need to do is to provide a special function that can park the secondary cores until some event occurs (an interrupt perhaps, or a hardware register changing value), and a kmain_secondary function for the secondary cores to execute.

I have a simple program where Core 0 will:

  • Configure and Enable the MMU
  • Invalidate and then enable the L1 I Cache and L1 D Cache
  • Invalidate and then enable the L2 Cache
  • Mark itself as being in "SMP" mode in the Auxilliary Control Register (ACTLR)
  • Enable the Snoop Control Unit and invalidate the SCU entries for Core 0
  • Map the Boot ROM back in at address 0x0
  • Do some UART logging whilst it does all that
  • Talk to the SoC's reset manager peripheral to take Core 1 out of reset
  • Wait for a global shared AtomicBoolto read astrue

Core 1 will:

  • Configure and Enable the MMU (each processor has its own)
  • Invalidate and then enable the L1 I Cache and L1 D Cache
  • Mark itself as being in "SMP" mode in the Auxilliary Control Register (ACTLR)
  • Invalidate the Snoop Control Unit entries for Core 0
  • Do some UART logging whilst it does all that
  • Sets the global shared AtomicBooltotrue

This works! Right up to that last bit - Core 1 is running, and it sets that flag to true. But Core 0 never observes the value changing from false to true. The two cores are seeing entirely different values for the same memory address, which is not supposed to happen (and will wreck pretty much any SMP system).

Whilst the two processors do have their own L1 caches, they also share a thing called the Snoop Control Unit (or SCU). This basically is a piece of hardware that watches what goes in and out of each of the processors and invalidates the other processors caches. This means the processors share a "coherent" view of the world, even if the write from one processor hasn't fully made it out to SDRAM yet (because it's still in L1 or L2 cache).

It should be very simple - you invalidate the SCU's entries and turn it on, and then magically it all works. OK, well it's not that simple because this copy and this copy of the Arm Cortex-A9 Technical Reference Manual disagree on whether you should set the SCU_CTRL.EN bit to 1 to enable it, or to 0 to enable it. But either way, it's not working (Linux sets it to 1, for what it's worth).

I've spent countless hours going over Arm documentation for both Armv7-A architecture, and the Cortex-A9 in particular. I've looked at example C code for the Altera Cyclone-V (in FreeRTOS and in ThreadX, and in Altera's own driver libraries that both those RTOSes use). I've carefully ported over Altera's drivers for the SCU, the L2C-310, and the L1 I/D Caches, and the two cores are not cache coherent. And I've got no idea what I did wrong.

The code is at https://codeberg.org/thejpster/hello-de0-nano-soc and if you can find what I did wrong and tell me how fix it, I'll happily write a follow-up blog post about where my mistake was and how you were smart enough to find what I couldn't.