CARME
CARME · Testing and CI
Preview

Every test, every board, every commit. No hardware rack.

A board farm is slow, flaky and expensive, and it never has the board you need when you need it. CARME runs the same firmware on emulated boards, as many as your CI has cores, and gives the same answer every time.

carme-test tests/ -j 32running
  • runtest_beacon_seen_by_gateway
  • ···test_node_survives_brownout
  • ···test_aes_gcm_vectors
  • ···test_ota_power_cut
  • ···test_linux_boot_from_snap
  • ···test_can_bus_off_recovery
  • ···test_watchdog_reset
0 passed · 0 failed · 0.00sresults.xml
Tests are Python

Boot a board, drive its inputs, check what it does.

A .cmtest is a pytest-style file. Check UART output, registers, memory, network packets, GPIO, power state, or signals in RTL.

tests/ble_beacon.cmtest
from carme import system
def test_beacon_seen_by_gateway():
s = system("systems/ble_mesh.cmsystem") # a gateway and 12 nodes
s.load("node*", "build/node.elf")
s.load("gateway", "build/gateway.elf")
s["gateway"].uart("uart0").expect("node 12 joined", timeout_s=5)
def test_node_survives_brownout():
s = system("systems/ble_mesh.cmsystem")
s.load("node*", "build/node.elf")
s["node3"].power.brownout(at_ms=1200, for_ms=3)
s["node3"].uart("uart0").expect("recovered from brownout")
Built for CI

Same inputs, same result. No retries.

  • Parallel

    carme-test -j N across cores; shard across machines with --shard i/n

  • Deterministic

    Same inputs, same result. No retries to hide a flaky board

  • Fast starts

    Start every test from a .cmsnap saved after boot

  • Reports

    JUnit XML, HTML, coverage, and a trace for every failure

  • Replay

    Every failure prints the one command that replays it, bit for bit

  • Any CI

    A single command in GitHub Actions, GitLab CI, Jenkins or Buildkite; runs in a container

Tests the hardware cannot run

Faults, time and scale on demand.

  • Faults on demand:corrupt a flash page, flip a bit in RAM, drop a CAN frame, stall the bus, glitch a clock, cut the power mid-write.

  • Time travel:run an hour of uptime in seconds, or stop time while you look.

  • Edge cases:the sensor at its limit, the radio at the edge of range, the battery at 3% and falling.

  • Scale:a thousand nodes on one network, each running its own firmware.

Coverage and reach

From the first commit to the field.

01

Coverage without instrumentation

CARME records which lines of the firmware each test reached, from the run itself, with no instrumented build. Merge it across the whole suite and see which code no test has ever executed.

02

One suite, every stage

The tests that run on emulated boards in CI also run against RTL in Lumina Sim and, with the same files, against an FPGA prototype or the real board. One suite follows the product from the first commit to the field.

Questions

Common questions.

Can I reuse my existing tests?

Tests that talk to the board over UART, network or a debugger work as they are: CARME exposes the same interfaces. Python test code moves to .cmtest with a few changes.

How many tests can run at once?

One per host core is typical; a microcontroller test needs tens of megabytes. Linux-class machines need more, and fewer run per host.

Do results depend on the host?

No. A test gives the same result on a laptop, a CI runner and a server.

Bring the test your board farm keeps failing.

Privacy settings

Choose what this site may use. You can change your mind at any time from the Privacy settings link at the bottom of every page.

What the site needs to work, and the record of the choice you make here, kept in your browser. No cookies are set.

Always on

Would let us count visits and see which pages are useful, without advertising or cross-site tracking. We do not use any analytics today, so nothing is loaded whichever you choose.

Cookie policy · Privacy policy