We'd be a lot more sane if we only supported workers in two clouds for now, by dropping packet.net.

In the absence of this, we need to scramble to build out packet.net support in new staging and production releng deployments.

Current status is that I'm able to boot: instances based off of the Ubuntu 18.04 AMI w/ 4.15.0-1045-aws kernel successfully, but that kernel was compiled without support for videodev, so our V4L2 based video loopback devices don't work.

modprobe videodev
modprobe: FATAL: Module videodev not found in directory /lib/modules/4.15.0-1045-aws

dmesg output:

[ 1342.150389] v4l2loopback: Unknown symbol video_ioctl2 (err 0)
[ 1342.150401] v4l2loopback: Unknown symbol v4l2_get_timestamp (err 0)
[ 1342.150417] v4l2loopback: Unknown symbol v4l2_ctrl_handler_init_class (err 0)
[ 1342.150439] v4l2loopback: Unknown symbol video_devdata (err 0)
[ 1342.150454] v4l2loopback: Unknown symbol v4l2_ctrl_new_custom (err 0)
[ 1342.150467] v4l2loopback: Unknown symbol video_unregister_device (err 0)
[ 1342.150479] v4l2loopback: Unknown symbol video_device_alloc (err 0)
[ 1342.150493] v4l2loopback: Unknown symbol v4l2_device_register (err 0)
[ 1342.150511] v4l2loopback: Unknown symbol __video_register_device (err 0)
[ 1342.150522] v4l2loopback: Unknown symbol v4l2_ctrl_handler_free (err 0)
[ 1342.150532] v4l2loopback: Unknown symbol v4l2_device_unregister (err 0)
[ 1342.150542] v4l2loopback: Unknown symbol video_device_release (err 0)

modprobe v4l2loopback
modprobe: ERROR: could not insert 'v4l2loopback': Unknown symbol in module, or unknown parameter (see dmesg)

I've also experimented with the generic kernel, which is what we've used successfully for other EC2 instance types running docker worker, and I've been able to boot an AMI with that kernel successfully. The kernel includes the videodev kernel module via extras, but won't boot on metal instances.

My current plan is to investigate into the following and pursue whichever is easier:

• getting AMIs based off of the generic kernel to boot on metal instances (need to figure out why they won't pass EC2 status checks and cannot be connected to)
• getting AMIs based off of the aws kernel to be able to run the videodev kernel module

Essentially, v4l2loopback needs videodev to be loaded before it is. Not having a kernel compiled with videodev throws a wrench in that.

It should be possible to compile videodev using DKMS, so that is a potential option for getting videodev running on the aws kernel.

We've figured out our kernel issues and are able to boot the generic kernel on AWS metal instances. Coop and Wander have taken a look at running tasks on metal instances in bug 1578460.

I'm going to close this (as it is more relevant to configuring worker instances) in favor of their bug.