diff --git a/diag.getxt b/diag.getxt index 84edff9..53293c3 100644 --- a/diag.getxt +++ b/diag.getxt @@ -1,8 +1,10 @@ -[ Internet ] { flow: north } -- [ PoE/VLAN Switch ] -[ Host machine ] -- [ PoE/VLAN Switch ] -[ Host machine ] { flow: south } -- [ USB Hub ] +[ Internet ] { origin: PoE/VLAN Switch; offset: 0, -2 } -- [ PoE/VLAN Switch ] +[ Controller ] -- { end: west } [ PoE/VLAN Switch ] +[ Controller ] { flow: south } -- [ USB Hub ] [ PoE Splitter ] { flow: north } -- [ PoE/VLAN Switch ] -[ RockPro64 ] -- { label: "Network" } [ PoE Splitter ] -[ RockPro64 ] -- { label: "Power" } [ PoE Splitter ] +[ Board ] -- { label: "Network" } [ PoE Splitter ] +[ Board ] -- { label: "Power" } [ PoE Splitter ] [ USB Hub ] -- [ Serial adapter ] -[ Serial adapter ] { flow: south; } -- [ RockPro64 ] +[ USB Hub ] { flow: north } -- [ SDWire ] { origin: USB Hub; offset: 0, 2 } +[ SDWire ] -- [ Board ] +[ Serial adapter ] { flow: south; } -- [ Board ] diff --git a/lab.doc.md.html b/lab.doc.md.html index 44a0188..1355cba 100644 --- a/lab.doc.md.html +++ b/lab.doc.md.html @@ -1,25 +1,40 @@ -Embedded Board dev cluster -========================== +**Embedded Board Dev Lab** -This is the description of how the cluster is architected and setup. There +Overview +======== + +This is the description of how the lab is architected and setup. There are a few design decisions that are likely to be different in other -environments, but this made the most sense for mine. +environments, but hopefully the overall architecture will be similar. +The differences should be laid the documentation for their instances. Goal ---- -The goal of this setup is to allow people to be able to install and test -images on various boards remotely. - -### Definitions - -- User - A person or entity that is able to reserve a system, and test with it. -- Host system - The machine that controls and provides access to the systems. - -### Features +The goal of this setup is to allow users to be able to install and test +FreeBSD patches and images on boards that they do not have locally. + +Definitions +----------- + +- Board - The system that is available for testing. This can be an SBC, + another computer, or any other system that needs to be tested. +- Board class - A set of boards of similar type. +- Board Handle - An identifier for a specific board instance. It will be + consistent until the lab is reconfigured. It can be used to help debug + issues that affect a specific board, or reserve a specific board that + has special hardware, such as an SDWire attached to it. +- Controller - The machine that controls and grants access to the boards. +- Jail - A jail on the controller that the User is able to log into. It + has access to the board, such as serial and ethernet. It will also be + able to power cycle the board. +- User - A person or entity that is able to reserve a board, and use it. + +Features +-------- - Remote power control - serial console access @@ -30,135 +45,154 @@ images on various boards remotely. - Long term goals: - Emulated microSD cards -### Physical Architecture - -The following diagrams the connections between the components. It is expected -that the connections for the RockPro64 is followed similarly for other embedded -devices. - -In the case of the Switch, the RockPro64 will be put on a VLAN which will be -delivered to the Host machine tagged. This will allow a jail in the host -machine to have a direct control over the broadcast domain for the device. -This will allow running dhcp/bootp/tftp services for netbooting by running dnsmasq -or another service. As fusefs is now jail friendly, the root FS could even be -mounted via sshfs - -The PoE part of the switch will be used for power control. PoE splitters (~$10) -are readily available and inexpensive, and the cost per port is realatively -inexpensive considering that power consumption will also be provided. - -************************************************************************** -* * -* +--------------+ +-----------------+ * -* | Host machine +-------------------------+ | Internet | * -* +-+------------+ | +-+---------------+ * -* | | | * -* | | | * -* | | | * -* +-+------------+ +----------------+ | +-+---------------+ * -* | USB Hub +-----+ Serial adapter | +---------+ PoE/VLAN Switch | * -* +--------------+ +-+--------------+ +-+---------------+ * -* | | * -* | | * -* | | * -* +-+--------------+ Network +-+---------------+ * -* | RockPro64 +------------+ PoE Splitter | * -* +-+--------------+ +-+---------------+ * -* | Power | * -* +-----------------------------+ * -* * -************************************************************************** - -### Logical Architecture - -No user will be able to log into the host machine directly. The only user -interface exposed on the host will be via an RPC interface, e.g. -`echo function xxx | ssh labhost labcli`, or via a socket from within -the jail. - -The user will be able to log into the jail that is created during the -reservation. Any modifications to the jail will be rolled back and -discarded after the system has been released, or the reservation has -expired. +Physical Architecture +--------------------- + +The following diagrams the connections between the components. There will +be many different instances of the board. + +The switch will have a unique VLAN assigned to each board. The controller +will receive the VLANs tag to deliver them to the correct jail. This +allows a jail on the host machine to have a direct control over the +broadcast domain for the device. It will allow running dhcp/bootp/tftp +services for netbooting. As fusefs is jail friendly, the root FS could +even be mounted via sshfs and exported to the board via NFS. + +The PoE part of the switch will be used for power control. PoE splitters +(~$10) are readily available and inexpensive, and the cost per port is +realatively inexpensive. The switch will be able to provide granular power +consumption on a per board basis. + +Some instances of the board will use an +[SDWire](https://wiki.tizen.org/SDWire). The SD reader will be accessible +in the jail, and a mechanism will be provided to switch the microSD card +between the jail and the board. This will allow easier testing of images. + +************************************************************************ +* * +* +------------+ +-----------------+ * +* | Controller +-------------------------+ | Internet | * +* +-+----------+ | +-+---------------+ * +* | | | * +* | | | * +* | | | * +* +-+----------+ +----------------+ | +-+---------------+ * +* | USB Hub +-----+ Serial adapter | +---------+ PoE/VLAN Switch | * +* +-+----------+ +-+--------------+ +-+---------------+ * +* | | | * +* | | | * +* | | | * +* +-+----------+ +-+--------------+ Network +-+---------------+ * +* | SDWire +-----+ Board +------------+ PoE Splitter | * +* +------------+ +-+--------------+ +-+---------------+ * +* | Power | * +* +-----------------------------+ * +* * +************************************************************************ + +Logical Architecture +-------------------- + +No user will be able to log into the controller directly. The only +interface exposed to the user on the controller will be via an RPC +interface, e.g. `echo function xxx | ssh labhost labcli`, or via a control +socket from within the jail. + +The user will be able to log into the jail that is created for the +reservation of a board. Any modifications to the jail will be rolled +back and discarded after the system has been released, or the reservation +has expired. A persistant user directory will be mounted and shared +between all jails, i.e. if the user has three different boards reserved, +each jail will have a nullfs mount of the user's directory. Workflow --------- +======== -# Functions +Functions +--------- -The functions that take a device handle can be executed from within the device jail -and the device handle of that device jail will be used. An error will be raised if -a device handle is provided and it does not match the current jail. +The functions that take a board handle argument can be executed from within +the jail and the board handle of that jail will be used. The board handle +argument will be ignored and is optional when used from withing the jail. These are the functions a user can execute: -1. List device classes (onlyavailable=false) - List the device classes that are known about. If onlyavailable is true, than only - ones that are currently available for claiming are returned. -2. Device status (claimed=false) - List all the device statuses currently. This includes that status, claimed or - unclaimed. If the device is claimed, it includes the user to claimed it. If claimed - is true, only list devices that are currently claimed by you. The default is to list - all devices. -3. Claim device (device class, power=false) - A user can use this to lock a device. This will return a device handle with the - device information, such as the device's jail's IP address, or an error. Once they - have obtained a device, it will not be allocated to another user till they have release - this device (or in the future a timeout has been hit). The user's ssh keys will be - automatically populated in that jail. By default, the device will be in the power off - state. When a default configuration is provided for the board, it can be automatically - powered on to make it easier to integrate w/ systems like CI. -4. Reinit device (device handle) - This will remove the current jail, and recreate it as if it was claimed for the first - time. This can be done so you can get the jail in a clean state w/o risk losing the - lock by freeing it and then reclaiming it. If you have not claimed the device, an - error will be returned. If the reinit fails, an error will be returned, but the claim - will be maintained. -5. Release device (device handle) - Release a claim on the device handle returned by claim device. This will make it - available to users again. All data in the jail will be deleted. It will return an - error if you do have have a claim on the device. -6. Power off (device handle) - Turn off the power to the device. -7. Power on (device handle) - Turn on the power to the device. - -# Services provided in the device jail +1. List board classes (onlyavailable=false) + List the board classes that are on the controller. If onlyavailable is + true, than only the classes that have boards available for reservation + are returned. +2. Board status (reserved=false) + List all the board statuses. This includes the status, reserved or + unreserved. If the board is reserved, it includes the user who reserved + it. If reserved is true, only list boards that are currently reserved + by you. The default is to list all boards. +3. Reserve board (board class or board id, power=false) + A user can use this to reserve a board. This will return a board handle + with the board information, such as the board's jail's IP address, or an + error. Once the user has obtained a board, it will not be allocated to + another user till they have release this board (or a timeout has + expired). The user's ssh keys will be automatically populated in the + jail. By default, the board will be in the power off state. When a + default configuration is provided for the board, it can be automatically + powered on, via the power arugment being true to make it easier to + integrate w/ systems like CI. + It is expected that in the future, additional arguments will allow the + user to specify different environments (e.g. stable/12). +4. Reinit board (board handle) + This will kill all processes and remove the current jail. The jail will + be recreated as if it was reserved for the first time. This allows the + jail to be in a clean state w/o losing the reservation by releasing it + and then reclaiming it. This will be important if there are pending + reservations for the board. If the user has not claimed the board, an + error will be returned. If the reinit fails, an error will be returned, + but the claim will be maintained if possible. +5. Release board (board handle) + Release a reservation on the board handle returned by reserve board + function. This will make the board available to users again. All data + in the jail will be deleted. The function will return an error if the + user does not have a claim on the board. +6. Power off (board handle) + Turn off the power to the board. +7. Power on (board handle) + Turn on the power to the board. + +Services provided in the jail +----------------------------- Once logged in, the jail will have the following services: 1. ssh Incoming ssh must be provided for the user to login. 2. One interface for internet nat setup so that all traffic appears from jail's IP. -3. One interface for device +3. One interface for board 4. Serial port for console access - The configuration will identify which device to put in the jail. For + The configuration will identify which board to put in the jail. For USB devices, they will be able to be specified via a list of ports on the hubs so that there is no issue w/ device probe order. -5. dhcpd for device w/ tftp already configured +5. dhcpd for boards w/ tftp already configured 6. inetd w/ tftpd setup 7. nfsd setup w/ exports configured -8. power control for device +8. power control for board -As the user will have root access, all of these can be modified after. +As the user will have root access, all of these can be modified. Future Work =========== There are a number of ideas to make developing boards remotely more doable. -We can use devices that support OTG, to be USB devices for test boards. This -means you can simulate a keyboard and or a mouse. With the combination of a -HDMI to ethernet adapter (there are cheap ones for ~$40/each), a developer can -work on making X or other GUI work remotely. +We can use devices that support OTG, to emulate USB devices for test boards. +This would allow the simulate a keyboard and or a mouse. With the addition +of an HDMI to ethernet adapter (there are cheap ones for ~$40/each), a user +can work on making X or other GUI work remotely. -Questions -========= +Open Questions +============== -Should the host key for the device be kept between invocations. -Pros: Users don't have to munch the known_hosts every time. +Should the jail's host key for the board be kept between invocations. +Pros: Users don't have to modify the known_hosts every time. Cons: Malicious user could impersonate the jail as they can copy out the key. IPv6 support? I have enough that each device can get a /64, but this remove privacy in - that the network will tell which device was active at the time + that the network will tell which board was active at the time