Information and documentation on building an embedded board test lab.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

199 lines

  1. <meta charset="utf-8">
  2. <meta http-equiv="refresh" content="2">
  3. **Embedded Board Dev Lab**
  4. Overview
  5. ========
  6. This is the description of how the lab is architected and setup. There
  7. are a few design decisions that are likely to be different in other
  8. environments, but hopefully the overall architecture will be similar.
  9. The differences should be laid the documentation for their instances.
  10. Goal
  11. ----
  12. The goal of this setup is to allow users to be able to install and test
  13. FreeBSD patches and images on boards that they do not have locally.
  14. Definitions
  15. -----------
  16. - Board - The system that is available for testing. This can be an SBC,
  17. another computer, or any other system that needs to be tested.
  18. - Board class - A set of boards of similar type.
  19. - Board Handle - An identifier for a specific board instance. It will be
  20. consistent until the lab is reconfigured. It can be used to help debug
  21. issues that affect a specific board, or reserve a specific board that
  22. has special hardware, such as an SDWire attached to it.
  23. - Controller - The machine that controls and grants access to the boards.
  24. - Jail - A jail on the controller that the User is able to log into. It
  25. has access to the board, such as serial and ethernet. It will also be
  26. able to power cycle the board.
  27. - User - A person or entity that is able to reserve a board, and use it.
  28. Features
  29. --------
  30. - Remote power control
  31. - serial console access
  32. - Network boot
  33. - Internet connectivity for boards w/ ethernet
  34. - Isolation between board environments
  35. Likely implemented via VLANs+jails w/ VNET to provide complete control
  36. - Long term goals:
  37. - Emulated microSD cards
  38. Physical Architecture
  39. ---------------------
  40. The following diagrams the connections between the components. There will
  41. be many different instances of the board.
  42. The switch will have a unique VLAN assigned to each board. The controller
  43. will receive the VLANs tag to deliver them to the correct jail. This
  44. allows a jail on the host machine to have a direct control over the
  45. broadcast domain for the device. It will allow running dhcp/bootp/tftp
  46. services for netbooting. As fusefs is jail friendly, the root FS could
  47. even be mounted via sshfs and exported to the board via NFS.
  48. The PoE part of the switch will be used for power control. PoE splitters
  49. (~$10) are readily available and inexpensive, and the cost per port is
  50. realatively inexpensive. The switch will be able to provide granular power
  51. consumption on a per board basis.
  52. Some instances of the board will use an
  53. [SDWire]( The SD reader will be accessible
  54. in the jail, and a mechanism will be provided to switch the microSD card
  55. between the jail and the board. This will allow easier testing of images.
  56. ************************************************************************
  57. * *
  58. * +------------+ +-----------------+ *
  59. * | Controller +-------------------------+ | Internet | *
  60. * +-+----------+ | +-+---------------+ *
  61. * | | | *
  62. * | | | *
  63. * | | | *
  64. * +-+----------+ +----------------+ | +-+---------------+ *
  65. * | USB Hub +-----+ Serial adapter | +---------+ PoE/VLAN Switch | *
  66. * +-+----------+ +-+--------------+ +-+---------------+ *
  67. * | | | *
  68. * | | | *
  69. * | | | *
  70. * +-+----------+ +-+--------------+ Network +-+---------------+ *
  71. * | SDWire +-----+ Board +------------+ PoE Splitter | *
  72. * +------------+ +-+--------------+ +-+---------------+ *
  73. * | Power | *
  74. * +-----------------------------+ *
  75. * *
  76. ************************************************************************
  77. Logical Architecture
  78. --------------------
  79. No user will be able to log into the controller directly. The only
  80. interface exposed to the user on the controller will be via an RPC
  81. interface, e.g. `echo function xxx | ssh labhost labcli`, or via a control
  82. socket from within the jail.
  83. The user will be able to log into the jail that is created for the
  84. reservation of a board. Any modifications to the jail will be rolled
  85. back and discarded after the system has been released, or the reservation
  86. has expired. A persistant user directory will be mounted and shared
  87. between all jails, i.e. if the user has three different boards reserved,
  88. each jail will have a nullfs mount of the user's directory.
  89. Workflow
  90. ========
  91. Functions
  92. ---------
  93. The functions that take a board handle argument can be executed from within
  94. the jail and the board handle of that jail will be used. The board handle
  95. argument will be ignored and is optional when used from withing the jail.
  96. These are the functions a user can execute:
  97. 1. List board classes (onlyavailable=false)
  98. List the board classes that are on the controller. If onlyavailable is
  99. true, than only the classes that have boards available for reservation
  100. are returned.
  101. 2. Board status (reserved=false)
  102. List all the board statuses. This includes the status, reserved or
  103. unreserved. If the board is reserved, it includes the user who reserved
  104. it. If reserved is true, only list boards that are currently reserved
  105. by you. The default is to list all boards.
  106. 3. Reserve board (board class or board id, power=false)
  107. A user can use this to reserve a board. This will return a board handle
  108. with the board information, such as the board's jail's IP address, or an
  109. error. Once the user has obtained a board, it will not be allocated to
  110. another user till they have release this board (or a timeout has
  111. expired). The user's ssh keys will be automatically populated in the
  112. jail. By default, the board will be in the power off state. When a
  113. default configuration is provided for the board, it can be automatically
  114. powered on, via the power arugment being true to make it easier to
  115. integrate w/ systems like CI.
  116. It is expected that in the future, additional arguments will allow the
  117. user to specify different environments (e.g. stable/12).
  118. 4. Reinit board (board handle)
  119. This will kill all processes and remove the current jail. The jail will
  120. be recreated as if it was reserved for the first time. This allows the
  121. jail to be in a clean state w/o losing the reservation by releasing it
  122. and then reclaiming it. This will be important if there are pending
  123. reservations for the board. If the user has not claimed the board, an
  124. error will be returned. If the reinit fails, an error will be returned,
  125. but the claim will be maintained if possible.
  126. 5. Release board (board handle)
  127. Release a reservation on the board handle returned by reserve board
  128. function. This will make the board available to users again. All data
  129. in the jail will be deleted. The function will return an error if the
  130. user does not have a claim on the board.
  131. 6. Power off (board handle)
  132. Turn off the power to the board.
  133. 7. Power on (board handle)
  134. Turn on the power to the board.
  135. Services provided in the jail
  136. -----------------------------
  137. Once logged in, the jail will have the following services:
  138. 1. ssh
  139. Incoming ssh must be provided for the user to login.
  140. 2. One interface for internet
  141. nat setup so that all traffic appears from jail's IP.
  142. 3. One interface for board
  143. 4. Serial port for console access
  144. The configuration will identify which board to put in the jail. For
  145. USB devices, they will be able to be specified via a list of ports on
  146. the hubs so that there is no issue w/ device probe order.
  147. 5. dhcpd for boards w/ tftp already configured
  148. 6. inetd w/ tftpd setup
  149. 7. nfsd setup w/ exports configured
  150. 8. power control for board
  151. As the user will have root access, all of these can be modified.
  152. Future Work
  153. ===========
  154. There are a number of ideas to make developing boards remotely more doable.
  155. We can use devices that support OTG, to emulate USB devices for test boards.
  156. This would allow the simulate a keyboard and or a mouse. With the addition
  157. of an HDMI to ethernet adapter (there are cheap ones for ~$40/each), a user
  158. can work on making X or other GUI work remotely.
  159. Open Questions
  160. ==============
  161. Should the jail's host key for the board be kept between invocations.
  162. Pros: Users don't have to modify the known_hosts every time.
  163. Cons: Malicious user could impersonate the jail as they can copy out the key.
  164. IPv6 support?
  165. I have enough that each device can get a /64, but this remove privacy in
  166. that the network will tell which board was active at the time
  167. <!-- Markdeep: --><style class="fallback">body{visibility:hidden;white-space:pre;font-family:monospace}</style><script src="markdeep.min.js" charset="utf-8"></script><script src="" charset="utf-8"></script><script>window.alreadyProcessedMarkdeep||("visible")</script>