Always Read The Documentation
ConnectX-3 VLAN Limitations: Proxmox and "no space left on device"
The 'no space left on device' error
If you're running an older Mellanox ConnectX-3 card in a Proxmox server (or any Linux box really) and when you restart the networking service or run ifreload -a you've seen RTNETLINK answers: No space left on device . It doesn't mean you've run out of disk space (or that ifreload thinks you have) it means that the ConnectX-3 card has reached it's hardware limitation of how many VLANs you can tag on one interface.
This is an issue that led to me feeling like i was going insane for almost a year and assuming that some of my VLANs just "didn't work" or that I'd somehow done something wrong elsewhere in the network, but no, my network is perfect as always.
It was only when i had the motivation to go digging to sort that mysterious "why do some VLANs not work" and "whats that error that appears when i restart the networking service" problem that I actually took note of the "No space left on device" error instead of dismissing it as a "my Proxmox server is making things up again" issue.
After a few hours of "no space left on device means your drive is full" forum threads I found a few threads discussing a 128 VLAN limit, which is backed up on the spec sheet for the card from Mellanox. Always read the documentation!
Why only 128 VLANs?
According to Mellanox's own spec sheet for the ConnectX-3 card: https://www.telecomdatasheets.com/ddata/539.pdf (Page 2, under Ethernet) the card only supports 128 MAC addresses per port:
This is likely due to a hardware limitation with a chipset used onboard these budget-friendly 10GbE cards rather than a driver issue as the spec sheet implies it's an OS agnostic issue.
One forum user who got in touch with Mellanox support was told that some older Linux kernels where not able to report that there was an issue with your VLAN definition and would fail silently. This is no longer the case, but the error message is still confusing.
The Linux driver's instruction manual (Page 56-57): https://manualzz.com/doc/25897014/mellanox-connectx-4-lx---connectx-3-pro---connectx-2-netw...?&p=56 states the following on the topic:
The important part: Each Virtual Interface / Physical Interface (VF/PF) can ask for up to 128 MACs on the policy of first-asks first-served.
This means that each physical interface can only have up to 128 VLAN subinterfaces configured on it, there is no restriction on the VLAN tags numbers themselves, only how many VLAN tags you can have.
How This Issue Presents Itself
In a Linux environment (most commonly proxmox or other virtualisation platforms) the error presents as follows:
- You allow more than 128 VLANs to traverse an interface under
bridge-vidsin/etc/network/interfaces - Upon running
ifreload -aor otherwise restarting networking you see something similar to this fill your terminal:
Solution
The solution to this issue is very simple, simply manually define your allowed VLANs individually in /etc/network/interfaces as shown below
And remember that you can only have 128 different VLANs traversing over each physical interface
