A Serial/IPMI interface would be much more suited for kernel debugging.

This has been on my extended todo list for a long time now, but there were always higher priority issues. I'm glad someone is working on it!

PR https://github.com/vyos/vyos-1x/pull/446
Note: gw-ip-address necessary define for [radius] or [chap-secrets] sections.

@c-po I have not tried this previously, but if it works well, I would like to keep it for kernel debugging on bare-metal devices.

Overview of the effort:

T2551 has now been fixed and confirmed working.

@dmbaturin @syncer @jestabro @thomas-mangin @UnicronNL

It will be in the next rolling (any rolling newer then vyos-1.3-rolling-202006070117-amd64.iso)

Ah, nice! I will update to the rolling release and give it is go soon.

@njh you have been right that command can be quiet handy, but instead of using a logfile we query "journalctl" for the appropriate logs.

Idea: do not generate empty node.def file if there is no <properties> statement.

@mrozentsvayg look at https://community.openvpn.net/openvpn/ticket/360 (this is documented in the code comment right above your change as well). OpenVPN on Linux in server mode with standard protocols doesn't listen on IPv6, just IPv4. We need to force it to bind to a IPv6 socket using these undocumented *6 protocols, then it'll listen on both IPv4 and IPv6. This wouldn't be necessary if OpenVPN listened on IPv6 with the default protocols or autodetected whether the local or remote IPs are v4 or v6 and chose the correct socket type, but it doesn't. Complain to the above ticket 360. We're just working within the limitations imposed by OpenVPN.

In T2550#66208, @jjakob wrote:

What happens when one remote-host is IPv4 and one is IPv6? The proposed fix would leave the protocol as udp6 in that case and the error would still be there.

I agree with the downside of vyos-config-debug; it was provided to allow some analysis when other was not available. What was done before was a straight call to the vyatta backend; what is done now is a essentially what you're suggesting, but in a case where the context of config/configtree is not successfully initialized, hence the failure. This is a good point for analysis, however, and I will restore the behaviour, likely by (a) using a straight call to the backend, and meanwhile (b) investigating if we can have partial context for config/configtree in this case.

Can you ask the user if you want to start the migration failure fallback mechanism on the first boot of the new image when upgrading, and if the user chooses to enable this mechanism, you should let the user select an old secure image (execute the mechanism only on the first boot)?

Therefore when modifying the default boot image (either via 'add system image' or 'set system image default-boot') the script that modifies grub.cfg needs to add its own name to it (easiest to do by adding a comment line to grub.cfg with its menu entry number), so that the failsafe reboot knows into which image to reboot into (it parses out the entry number from the grub comment and sets the default entry to it).

Can you ask the user if you want to start the migration failure fallback mechanism on the first boot of the new image when upgrading, and if the user chooses to enable this mechanism, you should let the user select an old secure image (execute the mechanism only on the first boot)?

I think 'configure; load; commit' is important to make debugging easier and faster. There are issues with the vyos-config-debug method: it needs a full reboot to test every change, which can take minutes (and one may not fix the bug in the first 10 tries even, depending on how sleep deprived one is) and it lacks a easy way to see the scripts stdout/stderr (it is discarded unless we enable airbag's debug log, which is yet another thing to have in mind) as the standard traceback that's logged to /tmp may not be enough to catch the exact error and we need to print some variables to look at them or something like that. But mainly the issue is rebooting is much slower than just doing load/commit.
Can't we just read config.boot into the session config of configtree or config? Wasn't that exactly what was done before? I'm 100% there was a function that read the config.boot file in config.py in case the config system wasn't initialised.

Right, that's obvious. The issue is that we need to know to *which* image to switch to, but it's easy to solve, as I'll describe.

In fact, you may be able to switch the default startup item by simply trying to call the following command, and this corresponds to the following command:

I look forward to your success.

Ah right, then all the things about replacing config.boot aren't necessary. I was thinking that /config was permanent between images, I don't know how I forgot that it lives inside each image separately.

The now standard method of debugging was clarified in T2409, with reference to this ticket (vyos-config-debug); the question here is whether there is any way to use 'configure; load; commit' on failed boot config load --- in the worst cases, no, as the CLI has not succssfully initialized, but will investigate. Changing to bug for evaluation.

You can ask the user before the upgrade operation if you want to perform the fallback on failure function on the first boot after the upgrade, and select an old image that is safe when the migration fails.

The way the current bootup commit works, (I'm not 100% certain, I may be wrong in some parts) is that it first looks if any components of the config require syntax migration, if yes, it backs up config.boot to config.boot.pre-migration-$(date) and goes through the migrator scripts, each one modifying the configuration to bring the syntax up to date, then it saves the new component version string and new config into config.boot, and then commits it. Therefore the config.boot is already migrated to the new syntax even if the commit then fails, meaning you can't simply reboot into the old image and expect things to work, as the old image expects the old config syntax, but the config was already migrated to the new one. We need to replace config.boot with its .pre-migration version before rebooting into the old image.

Yes, let me confirm a few details here ....

Sorry, I don't know if I understand it wrong, but please allow me to express my opinion, but when you install and add a new image from the old image for upgrading, it may occur that it can't be used normally at startup config.boot Configuration (especially rolling update),

@jjakob Thanks for a good summary of one of the issues, namely, not replacing config until 'success' (at least of boot, if not other criteria). Migration does save a backup, but we had recently discussed not doing in place change by default. I would like to assess this in relation to other recent discussions of failsafe and rollback, and then proceed with that compatible change first ...

Please test with the latest 1.3 rolling image if the bug is now fixed.

Perhaps a good way is still for the user to choose whether to reconfigure or restart with the old image.

Sorry, I don't know if I understand it wrong, but please allow me to express my opinion, but when you install and add a new image from the old image for upgrading, it may occur that it can't be used normally at startup config.boot Configuration (especially rolling update), in which case, whether or not the old config.boot There is no intention to make any migration or modification Justice. Either the configuration syntax has been changed and errors still occur after migration, or some settings of the system image have problems. A better solution is to modify grub's default boot so that it can boot from the old normal system image, if it exists, but this requires user authorization.

@richardpowellus Yes, your work around fixed my problem and I'm now for the first time able to receive a /56 from my ISP and divvy it up between my interfaces.

Fixed in the rolling release from 20200606xx

@richardpowellus I've noted the exact same issue in T2551.

Richardpowellus: I may be running into the same issue. What does your
id-assoc pd 2 { } section look like?

I seem to have figured out my problem. It seems like setting "set interfaces ethernet eth1 dhcpv6-options prefix-delegation length 60" is not working (eth1 is my WAN interface that's connected directly to my cable modem in bridge mode).

Yes, we need to try/except the apply section (the other should never fail but we could still catch errors to not leave the system in an unknown state) but when applying the reverse configuration (ie: invert effective and new and re-apply) one must then be careful if that fails too (we do not want a forever loop :p). The code already runs all the get_dict and all the verify first, so we will only apply if all is ok, but still issues could occur.

In T421#66055, @gadams wrote:

@dsummers I have been able to get the current nightly builds to work on Comcast Business, which is delivered via ethernet. In this particular case, there are some unfortunate gotchas to keep in mind, but no modification of VyOS is currently needed, at least in my case. Very cool!

For anyone else who comes across this (I have talked to a couple folks outside of this task, as well), here are the gotchas, the second one of which is Comcast-specific:

• You currently need to enable getting an IPv6 address on the ethernet interface, even though that's technically not required other than by the config syntax. Using parameters-only will prevent it from working, since WIDE dhcp6c will interpret that to mean that you also don't want delegated adresses.

In my experience, this causes only some small issues, but not failure. I am drafting a proposal to fix it, but it's slightly more complex than I guessed.

• You need to know that while the Comcast cablemodem receives a /56 from the network to delegate to you, it carves off the first (sla-id 0) /64 of that for the link between it and your router, and then only delegates out of the highest (/56 4-bit sla-id f) /60 to your router. This is odd, and I don't know why they do it that way. So, you need to know that you'll be getting a /60 delegated to you, and that you should use 4-bit SLAs.

I had to figure this out by looking at the DHCPv6 conversation on the wire some time ago.

The config syntax could help a lot here, but this seems to be limited a bit by WIDE dhcp6c's conf options.

In any event, here's a working config I'm using (well, simplified, but still enough to make it work):

interfaces {
    ...
    ethernet eth3 {
        address 10.1.11.3/24
        address dhcpv6
        description OUTSIDE-v6
        dhcpv6-options {
            prefix-delegation {
                interface br0 {
                    address 1
                    sla-id 1
                    sla-len 4
                }
                interface eth2.4 {
                    address 1
                    sla-id 4
                    sla-len 4
                }
                interface eth2.6 {
                    address 1
                    sla-id 6
                    sla-len 4
                }
            }
        }
        ...
    }
    ....
}
...
    router-advert {
        interface br0 {
            name-server 2603:....
            name-server 2603:....
            prefix ::/64 {
                valid-lifetime 2592000
            }
        }
    .... and so on for the other interfaces ...

(The address 1 bits aren't strictly necessary, but I like my routers having easily recignizable addresses when I'm looking over traffic flows and routes and so on.)

Happy v6 surfing!

This could raise potential new bugs if a firmware file was missed out for whatever reason. Please open a new Task if this happened to you. Running it in @kroy's lab showed no problems so far.

Yes, we need to try/except the apply section (the other should never fail but we could still catch errors to not leave the system in an unknown state) but when applying the reverse configuration (ie: invert effective and new and re-apply) one must then be careful if that fails too (we do not want a forever loop :p). The code already runs all the get_dict and all the verify first, so we will only apply if all is ok, but still issues could occur.

@jjakob control-R should be implemented. It is a feature I use too and expect it - just did not think it was worth a demo 0.0 release - there is plenty to improve with the code in the branch - it surely has bugs -as I said POC :-)

https://github.com/vyos/vyos-1x/pull/444

@thomas-mangin that's great if the POC already has the above - I'm on board with making it the replacement for vbash in that case. I can live without word jumping and line deletion (for now) if it has tab completion and history (I will miss reverse history search a lot as it doesn't search just the beginning of the line but the whole line for the pattern, I'm assuming prompt-toolkit just searches the beginning?). I do need to test it when I get some time to see if anything is still missing.

About rollback, i'm wondering about a try:expect loop around apply() that will catch faults and trigger a rollback() to restore old files etc.
The rollback won't be a 100% abort, because vyatta-cfg would not rollback subsystems that have allready been configured.. but we will get a pr. Subsystem rollback and thats a start :) to get a full rollback wee need to change the backend or the executor in the backend.

generate should make a backup of the previous file before generating the new one. It will then make it possible to create the rollback as a file move and service reload.

@jjakob the code can be installed on a router (using my vyos update tool - after running vyos setup router )

To add my 2 cents:

Another possibility would be to modify the VTunIf's bridgeable parameter when creating it. That wouldn't require a different name for tap and all the config migration that comes with it, but I don't know if it's possible.

I also don't think we can find out whether a interface is tun or tap from the up/down script (no parameter or environment variable seems to tell it this) so it'd be good if tun and tap had different names due to this too. It would require migrating the name of existing interfaces (from vtun to vtap if tunnel type is tap) under interfaces openvpn, interfaces bridge member, service router-advert, firewall, nat, possibly other places as well.

What happens when one remote-host is IPv4 and one is IPv6? The proposed fix would leave the protocol as udp6 in that case and the error would still be there.

Writing the config parsing, editing, committing and saving shouldn't be that hard.
Re-implementing completion with the current set of features would be very hard.

A workaround would be to simply call interfaces-ethernet.py from the bond script itself after it removes it from the bond.
The consequence of migrating bond member config from the interface node to the bond node, the same problem exists for bridges as well.

Also regarding bridging: the current VTunIf says vtun is bridgeable. I doubt bridging a tun interface is wanted as that's a purely L3 tunnel. As the bridge member config syntax places the members under the bridge interface, the bridge interface determines if a interface is bridgeable by looking at its class definition. Thus to make openvpn in tun mode not bridgeable and tap mode bridgeable, those would need to be 2 different classes with different interface names ('vtun' and 'vtap'?). A hackish way is possible by making the bridge code check the openvpn config directly, but I highly dislike hackish solutions. Even T2241 was a 'hackish' solution that was necessary due to a previous bridge syntax migration without thinking about the consequences of it (moving the bridge member config under the bridge code makes syntactical sense, but it requires hackish workarounds like T2241 with the curernt way the config system operates)

Looking at the above errors: