Also reported in T1830

In FRR 7.0.1 (VyOS 1.2.3) was some bug, due to which static routes were not updated (maybe, not in all cases or environments) after the next-hop state change. In VyOS 1.2.4 we use stable FRR 7.2, which processes this situation without problems. An example (key point from FRR debug log):
FRR 7.0.1:

Jan 13 15:29:51 vyos zebra[1041]: 0:10.230.230.0/30: Adding route rn 0x5612ea69d1f0, re 0x5612ea69d370 (type 2)
Jan 13 15:29:51 vyos zebra[1041]: 0:10.230.230.0/30: Redist update re 0x5612ea69d370 (type 2), old (nil) (type -1)
Jan 13 15:29:51 vyos zebra[1041]: 0:10.230.230.0/30: Adding route rn 0x5612ea69d490, re 0x5612ea69e110 (type 2)
Jan 13 15:29:51 vyos zebra[1041]: 0:10.230.230.0/30: Redist update re 0x5612ea69e110 (type 2), old (nil) (type -1)
Jan 13 15:29:51 vyos zebra[1041]: NHT processing check for zvrf default
Jan 13 15:29:51 vyos zebra[1041]: 0:10.230.230.1/32: Evaluate RNH, type 0
Jan 13 15:29:51 vyos zebra[1041]: 0:10.230.230.1/32: NH resolved over route 10.230.230.0/30
Jan 13 15:29:51 vyos zebra[1041]: 0:10.230.230.1/32: Notifying client static about NH
Jan 13 15:29:51 vyos zebra[1041]: 0:192.168.20.1/32: Evaluate RNH, type 0

FRR 7.2:

Jan 13 15:33:23 vyos zebra[1042]: 0:10.230.230.0/30: Adding route rn 0x5574620a18b0, re 0x5574620a1930 (connected)
Jan 13 15:33:23 vyos zebra[1042]: 0:10.230.230.0/30: Adding route rn 0x5574620a29b0, re 0x5574620a1850 (connected)
Jan 13 15:33:23 vyos zebra[1042]: 0:10.230.230.0/30 update_from_ctx(): no fib nhg
Jan 13 15:33:23 vyos zebra[1042]: 0:10.230.230.0/30 update_from_ctx(): rib nhg matched, changed 'true'
Jan 13 15:33:23 vyos zebra[1042]: 0:10.230.230.0/30: Redist update re 0x5574620a1930 (connected), old 0x0 (None)
Jan 13 15:33:23 vyos zebra[1042]: 0:10.230.230.1/32: Evaluate RNH, type Nexthop
Jan 13 15:33:23 vyos zebra[1042]: 0:10.230.230.1/32: NH resolved over route 10.230.230.0/30
Jan 13 15:33:23 vyos zebra[1042]: 0:10.230.230.1/32: Notifying client static about NH
Jan 13 15:33:23 vyos zebra[1042]: rib_add_multipath: 0:10.0.0.0/8: Inserting route rn 0x5574620a1b10, re 0x5574620a1a30 (static) existing (nil)
Jan 13 15:33:23 vyos zebra[1042]: 0:10.0.0.0/8: Adding route rn 0x5574620a1b10, re 0x5574620a1a30 (static)
Jan 13 15:33:23 vyos zebra[1042]: netlink_route_multipath(): RTM_NEWROUTE 10.0.0.0/8 vrf 0(254)
Jan 13 15:33:23 vyos zebra[1042]: netlink_route_multipath() (single-path): nexthop via 10.230.230.1  if 3(0)
Jan 13 15:33:23 vyos zebra[1042]: netlink_talk: netlink-dp (NS 0) type RTM_NEWROUTE(24), len=60 seq=10 flags 0x501
Jan 13 15:33:23 vyos zebra[1042]: 0:10.0.0.0/8 update_from_ctx(): no fib nhg
Jan 13 15:33:23 vyos zebra[1042]: 0:10.0.0.0/8 update_from_ctx(): rib nhg matched, changed 'true'
Jan 13 15:33:23 vyos zebra[1042]: 0:10.0.0.0/8: Redist update re 0x5574620a1a30 (static), old 0x0 (None)

So, configured static routes updating properly.

Dou can try to play around withe the node.def and interfaces-vxlan.py file and lower the limits to see if it works at all.

@elbuit

vyos@mpls# sudo nft list table nat
table ip nat {
	chain PREROUTING {
		type nat hook prerouting priority -100; policy accept;
		counter packets 0 bytes 0 jump VYATTA_PRE_DNAT_HOOK
	}

Thanks @Viacheslav, I've build a custom vyos 1.2.4 with nftables package because I want to add custom rules in nftables mode.

@elbuit
Rolling is support nft now.

vyos@mpls:~$ sudo nft -v
nftables v0.9.0 (Fearless Fosdick)

PR https://github.com/vyos/vyos-1x/pull/203
Adding commands for show mpls

All works, but seems we need works with timezone for correct converting USEC to "date time" format

Very Nice!

https://github.com/vyos/vyos-1x/commit/b69d20277b9625167b9d332775d8c7498dd8dd22

Has been fixed already.

Created new Azure image

I'm also affected by this, but even with a relatively "small" configuration (2662 lines, at present, where more than half are firewall rules, 5 interfaces).

First tests for MPLS.
Latest rolling releases is supported it.

In this case we can use /run/systemd/shutdown/scheduled file for detecting powertrl scheduler
PR for this task https://github.com/vyos/vyos-1x/pull/201

Hang on, I see the error in syslog (only syslog), so at least I can reproduce it now.

@Merijn I tested with the latest rolling and can't reproduce the issue. From your error message yesterday it looks like your snmpd.conf has an issue. Can you copy that content as well?

@jjakob https://downloads.vyos.io/rolling/current/amd64/vyos-1.3-rolling-202001081700-amd64.iso and later should address that issue.

@hagbard

set service snmp community dummycomm authorization 'ro'
set service snmp community dummycomm client '8.8.8.8'
set service snmp community dummycomm client '8.8.4.4'
set service snmp contact '[email protected]'
set service snmp location 'Datacenter, City, Country'

Successfully tested on latest rolling 1.3-rolling-202001080217

Confirmed fix with that commit.

https://github.com/vyos/vyos-1x/commit/3863e5dc24a622090f52c13f8aa04e5c9f00341a should fix it.
(http://dev.packages.vyos.net/repositories/current/pool/main/v/vyos-1x/vyos-1x_1.3.0-16_all.deb if you want to manually install and test or wait for tomorrows iso).

systemctl renamed it to syslog, so it won't be restarted correctly and the conf script won't generate the files correctly. It is correctly named within init.d.

It definitely remains in my config:

looks like service syslog did disappear from the default config.

@Merijn Can you please share the snmp part from your config?

I only have a workaround right now, it appears that grub.cfg can't be found. To boot at least the installed system do the following within the grub shell:

configfile /EFI/VyOS/grub.cfg

Move to 'Needs testing'; repo linked below. Add example of third-party package, and test, before PR.

Seems to be a simple fix so i've submitted a PR: https://github.com/vyos/vyos-1x/pull/199

@hagbard i tried testing by installing the package.
The service is running but not working correctly.
The following is shown:
Jan 07 10:25:54 server snmpd[9979]: /etc/snmp/snmpd.conf: line 10: Warning: Unknown token: smuxpeer.
Jan 07 10:25:54 server snmpd[9979]: /etc/snmp/snmpd.conf: line 11: Warning: Unknown token: smuxpeer.
Jan 07 10:25:54 server snmpd[9979]: /etc/snmp/snmpd.conf: line 12: Warning: Unknown token: smuxsocket.
Jan 07 10:25:54 server snmpd[9979]: notificationEvent OID: linkUp
Jan 07 10:25:54 server snmpd[9979]: /etc/snmp/snmpd.conf: line 21: Error: unknown notification OID
Jan 07 10:25:54 server snmpd[9979]: notificationEvent OID: linkDown
Jan 07 10:25:54 server snmpd[9979]: /etc/snmp/snmpd.conf: line 22: Error: unknown notification OID
Jan 07 10:25:54 server snmpd[9979]: /etc/snmp/snmpd.conf: line 23: Warning: Unknown token: monitor.
Jan 07 10:25:54 server snmpd[9979]: /etc/snmp/snmpd.conf: line 24: Warning: Unknown token: monitor.
Jan 07 10:25:54 server snmpd[9979]: net-snmp: 2 error(s) in config file(s)

Hi @bbabich
How we can reproduce this bug?
I tested with 55 bgp-sessions. Each with its own unique filter. All filters applied as needed.
Without filters I announced 111 routes.
With filters per session I export 1 route to each peer.

Sure thing, let me know the result.

@Merijn https://github.com/vyos/vyos-1x/commit/78df0c46865b3af89d6bc327b4c1d08cc4450aff or tomorrows rolling, as you seem to compile it yourself is should now work out of the box when you install the new vyos-1x package.
(http://dev.packages.vyos.net/repositories/current/pool/main/v/vyos-1x/vyos-1x_1.3.0-16_all.deb)

Debian default snmp user is called Debian-snmp while the script tries to get the uid of the user snmp. Looks like that is the entire issue.

PR for this task https://github.com/vyos/vyos-1x/pull/198

In 1.3 default-route 'force' works as expected, for 1.2.x (crux) we need merge patch which was proposed by @hagbard

This was tested on a different VyOS install, so I will test again and confirm.

I just did a build of Crux 1.2.4. The issue that appeared in 1.2.3 did not occur in 1.2.4.

Hi @rherold , I can't create this behaviour in LAB.
I have clear vyos-1.2.3 installed and do next steps:

1. sudo nano /config/scripts/vyos-postconfig-bootup.script

#!/bin/sh
# This script is executed at boot time after VyOS configuration is fully applie$
# Any modifications required to work around unfixed bugs
# or use services not available through the VyOS CLI system can be placed here.
/config/scripts/rcs/rcs-mgnt-vlan.sh 2>/dev/null 1>/dev/null

2. sudo mkdir /config/scripts/rcs/
3. Create file sudo nano /config/scripts/rcs/rcs-mgnt-vlan.sh with following commands

#!/bin/bash

Tested successfully on VyOS 1.3-rolling-202001060217

Hi @max1e6 , can you please share the relevant openvpn config, so I can try to reproduce the issue? Thx.

@kroy thx for testing, glad it is working for you since I'm not really satified with it.

@hagbard Confirmed your hack takes care of the issue.

Yup, https://phabricator.vyos.net/T1831 is pending. FRR RAs will fix the issue entirely or going with set service ipv6-ra interface... would fix that too regardless what daemin is being used at the backend which would also make it interchangeable.

I was hoping you have some input here.

As I can remember there is no objection about FRR for RAs - its only the CLI structure.

I actually suggested using reload/SIGHUP. The problem is the very rapid reloads sent by the vyos script to systemd. start-stop-daemon is handled by systemd in Debian Buster, in Jessie it was still handled by sysvinit so it didn't have any limits. I suppose it uses some default restart limits/timeouts that can otherwise be adjusted in unit/service files. I suppose it could be converted to a native systemd service so the limits can be set if there is a corresponding setting that would fix the issue. Otherwise it'd be better if we don't use systemd to send SIGHUP at all and send the signal direclty to the daemon w/ pid read from pid file. Or switch to using frr for RAs - what's the progress on that?

That's what it does but using the init script.