Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TAP devices and virtio sock for guest auraed #446

Open
krisnova opened this issue Mar 9, 2023 · 8 comments
Open

TAP devices and virtio sock for guest auraed #446

krisnova opened this issue Mar 9, 2023 · 8 comments
Assignees
Labels
Subsystem: Network Beeps and Boops #OnlyLANs

Comments

@krisnova
Copy link
Contributor

krisnova commented Mar 9, 2023

Given our recursive nature of hosting nested auraed instances in virtual guests, nested cells, and container isolation zones we need to start understanding the connection detail between the host and the guest.

This issue suggests we use the following strategy:

  • TAP devices on the host to consume ethernet frames
  • VirtIO sock on the guest to route to

By default we use IPv6 between the host and the guest, regardless of what the host is using. We take a strong opinion against v-sock implementation and exposing shared memory or file descriptor access to a guest.

@MalteJ
Copy link
Contributor

MalteJ commented Mar 9, 2023

Is this about the connection between auraed on the host system and auraed on the guest (MicroVM)?
There are two options:

  1. ethernet connection
  2. socket connection

With firecracker, an ethernet connection is probably the default connection between the host and a MicroVM. It was also the only option in the beginning, when AWS open sourced firecracker. For an ethernet connection, you need a tap device on a host. Via the Firecracker API this tap device will be connected to a virtio-net device, which is attached to the MicroVM. This virtio-net device will appear as a NIC in the VM.

A socket connection between host and MicroVM was added later on to firecracker. Probably it's not used by AWS in production. This works with virtio-vsock and provides a socket connection where one end resides on the host and the other end is within the VM. There are security concerns regarding this type of connection. I am no expert and I cannot explain what's the problem or possible attack vector. I just wouldn't use it.

So, I would vote for using an ethernet connection between the host and the VM. To connect to the nested auraed we can use IPv6 link-local addressing (fe80:: addresses). Those addresses will not be routed and therefore a connection can only be established locally. We can decide on well-known link local addresses to be assigned to auraed and nested auraed (e.g. fe80::1/64) or use stateless autoconfiguration and monitor the IPv6 neighbor discovery packets to find the addresses.

@MalteJ
Copy link
Contributor

MalteJ commented Mar 9, 2023

See also #433 (comment)

@MalteJ
Copy link
Contributor

MalteJ commented Mar 9, 2023

When you have a DPU that is controlling the host system, you will definitely have an Ethernet connection between the DPU and the host system. Some DPUs also provide an additional proprietary communication channel for control plane communication. But the common thing you will always have, are Ethernet connections.

From a security perspective I think Ethernet connections are fine. Lots of people invest lots of time to harden the Linux kernel's Ethernet stack.

@JeroenSoeters
Copy link
Contributor

In my experiments with cloud hypervisor I have also been using the TAP/virtio approach over IPv6 link-local.

Cloud-hypervisor has support for vhost-net as well (FC doesn't) so that is something we could look at as well: https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/device_model.md#vhost-user-net

@MalteJ MalteJ self-assigned this Jan 23, 2024
@MalteJ MalteJ added the Subsystem: Network Beeps and Boops #OnlyLANs label Jan 23, 2024
@bpmooch
Copy link
Contributor

bpmooch commented Jun 21, 2024

#506 works through some of this

@mccormickt
Copy link
Contributor

In my experiments with cloud hypervisor I have also been using the TAP/virtio approach over IPv6 link-local.

Cloud-hypervisor has support for vhost-net as well (FC doesn't) so that is something we could look at as well: https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/device_model.md#vhost-user-net

My implementation in #506 uses the default virtio-net (instead of vhost-user-net linked above) and it works great over link-local. I'm not as familiar with what will be needed for additional networking requirements, but I believe we'd need some additional configuration to allow workloads within the VM to communicate externally?

@bpmooch
Copy link
Contributor

bpmooch commented Aug 4, 2024

@mccormickt wasnt 100% clear, are you saying this ticket is completed per #506 ? minus the virio-net vs vhost-user-net nuance?

@mccormickt
Copy link
Contributor

mccormickt commented Aug 11, 2024

@mccormickt wasnt 100% clear, are you saying this ticket is completed per #506 ? minus the virio-net vs vhost-user-net nuance?

This is mostly completed. TAP devices are managed by cloud hypervisor for us and you can reach a nested auraed using ipv6 link-local.

By default we use IPv6 between the host and the guest, regardless of what the host is using.

We can currently only use IPv6 link local to communicate to the VM's auraed. Cloud hypervisor VMs also must be assigned an IPv4 address, so I'm not sure it satisfies the above. If we create the TAP devices ourselves, we can likely get around this limitation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Subsystem: Network Beeps and Boops #OnlyLANs
Projects
None yet
Development

No branches or pull requests

5 participants