fbsdembdev/lab.doc.md.html

<meta charset="utf-8">
<meta http-equiv="refresh" content="2">

**Embedded Board Dev Lab**

Overview
========

This is the description of how the lab is architected and setup.  There
are a few design decisions that are likely to be different in other
environments, but hopefully the overall architecture will be similar.
The differences should be laid the documentation for their instances.

Goal
----

The goal of this setup is to allow users to be able to install and test
FreeBSD patches and images on boards that they do not have locally.

Definitions
-----------

- Board - The system that is available for testing.  This can be an SBC,
  another computer, or any other system that needs to be tested.
- Board class - A set of boards of similar type.
- Board Handle - An identifier for a specific board instance.  It will be
  consistent until the lab is reconfigured.  It can be used to help debug
  issues that affect a specific board, or reserve a specific board that
  has special hardware, such as an SDWire attached to it.
- Controller - The machine that controls and grants access to the boards.
- Jail - A jail on the controller that the User is able to log into.  It
  has access to the board, such as serial and ethernet.  It will also be
  able to power cycle the board.
- User - A person or entity that is able to reserve a board, and use it.

Features
--------

- Remote power control
- serial console access
- Network boot
- Internet connectivity for boards w/ ethernet
- Isolation between board environments
  Likely implemented via VLANs+jails w/ VNET to provide complete control
- Long term goals:
  - Emulated microSD cards

Physical Architecture
---------------------

The following diagrams the connections between the components.  There will
be many different instances of the board.

The switch will have a unique VLAN assigned to each board.  The controller
will receive the VLANs tag to deliver them to the correct jail.  This
allows a jail on the host machine to have a direct control over the
broadcast domain for the device.  It will allow running dhcp/bootp/tftp
services for netbooting.  As fusefs is jail friendly, the root FS could
even be mounted via sshfs and exported to the board via NFS.

The PoE part of the switch will be used for power control.  PoE splitters
(~$10) are readily available and inexpensive, and the cost per port is
realatively inexpensive.  The switch will be able to provide granular power
consumption on a per board basis.

Some instances of the board will use an
[SDWire](https://wiki.tizen.org/SDWire).  The SD reader will be accessible
in the jail, and a mechanism will be provided to switch the microSD card
between the jail and the board.  This will allow easier testing of images.

************************************************************************
*                                                                      *
* +------------+                                   +-----------------+ *
* | Controller +-------------------------+         |    Internet     | *
* +-+----------+                         |         +-+---------------+ *
*   |                                    |           |                 *
*   |                                    |           |                 *
*   |                                    |           |                 *
* +-+----------+     +----------------+  |         +-+---------------+ *
* |  USB Hub   +-----+ Serial adapter |  +---------+ PoE/VLAN Switch | *
* +-+----------+     +-+--------------+            +-+---------------+ *
*   |                  |                             |                 *
*   |                  |                             |                 *
*   |                  |                             |                 *
* +-+----------+     +-+--------------+  Network   +-+---------------+ *
* |   SDWire   +-----+     Board      +------------+  PoE Splitter   | *
* +------------+     +-+--------------+            +-+---------------+ *
*                      |                Power        |                 *
*                      +-----------------------------+                 *
*                                                                      *
************************************************************************

Logical Architecture
--------------------

No user will be able to log into the controller directly.  The only
interface exposed to the user on the controller will be via an RPC
interface, e.g. `echo function xxx | ssh labhost labcli`, or via a control
socket from within the jail.

The user will be able to log into the jail that is created for the
reservation of a board.  Any modifications to the jail will be rolled
back and discarded after the system has been released, or the reservation
has expired.  A persistant user directory will be mounted and shared
between all jails, i.e. if the user has three different boards reserved,
each jail will have a nullfs mount of the user's directory.

Workflow
========

Functions
---------

The functions that take a board handle argument can be executed from within
the jail and the board handle of that jail will be used.  The board handle
argument will be ignored and is optional when used from withing the jail.

These are the functions a user can execute:
1. List board classes (onlyavailable=false)
   List the board classes that are on the controller.  If onlyavailable is
   true, than only the classes that have boards available for reservation
   are returned.
2. Board status (reserved=false)
   List all the board statuses.  This includes the status, reserved or
   unreserved.  If the board is reserved, it includes the user who reserved
   it.  If reserved is true, only list boards that are currently reserved
   by you.  The default is to list all boards.
3. Reserve board (board class or board id, power=false)
   A user can use this to reserve a board.  This will return a board handle
   with the board information, such as the board's jail's IP address, or an
   error.  Once the user has obtained a board, it will not be allocated to
   another user till they have release this board (or a timeout has
   expired).  The user's ssh keys will be automatically populated in the
   jail.  By default, the board will be in the power off state.  When a
   default configuration is provided for the board, it can be automatically
   powered on, via the power arugment being true to make it easier to
   integrate w/ systems like CI.
   It is expected that in the future, additional arguments will allow the
   user to specify different environments (e.g. stable/12).
4. Reinit board (board handle)
   This will kill all processes and remove the current jail.  The jail will
   be recreated as if it was reserved for the first time.  This allows the
   jail to be in a clean state w/o losing the reservation by releasing it
   and then reclaiming it.  This will be important if there are pending
   reservations for the board.  If the user has not claimed the board, an
   error will be returned.  If the reinit fails, an error will be returned,
   but the claim will be maintained if possible.
5. Release board (board handle)
   Release a reservation on the board handle returned by reserve board
   function.  This will make the board available to users again.  All data
   in the jail will be deleted.  The function will return an error if the
   user does not have a claim on the board.
6. Power off (board handle)
   Turn off the power to the board.
7. Power on (board handle)
   Turn on the power to the board.

Services provided in the jail
-----------------------------

Once logged in, the jail will have the following services:
1. ssh
   Incoming ssh must be provided for the user to login.
2. One interface for internet
   nat setup so that all traffic appears from jail's IP.
3. One interface for board
4. Serial port for console access
   The configuration will identify which board to put in the jail.  For
   USB devices, they will be able to be specified via a list of ports on
   the hubs so that there is no issue w/ device probe order.
5. dhcpd for boards w/ tftp already configured
6. inetd w/ tftpd setup
7. nfsd setup w/ exports configured
8. power control for board

As the user will have root access, all of these can be modified.

Future Work
===========

There are a number of ideas to make developing boards remotely more doable.
We can use devices that support OTG, to emulate USB devices for test boards.
This would allow the simulate a keyboard and or a mouse.  With the addition
of an HDMI to ethernet adapter (there are cheap ones for ~$40/each), a user
can work on making X or other GUI work remotely.

Open Questions
==============

Should the jail's host key for the board be kept between invocations.
Pros: Users don't have to modify the known_hosts every time.
Cons: Malicious user could impersonate the jail as they can copy out the key.

IPv6 support?
	I have enough that each device can get a /64, but this remove privacy in
	that the network will tell which board was active at the time

<!-- Markdeep: --><style class="fallback">body{visibility:hidden;white-space:pre;font-family:monospace}</style><script src="markdeep.min.js" charset="utf-8"></script><script src="https://casual-effects.com/markdeep/latest/markdeep.min.js" charset="utf-8"></script><script>window.alreadyProcessedMarkdeep||(document.body.style.visibility="visible")</script>