LEDE Project

  • Status Closed
  • Percent Complete
    100%
  • Task Type Bug Report
  • Category Kernel
  • Assigned To No-one
  • Operating System All
  • Severity High
  • Priority Very Low
  • Reported Version Trunk
  • Due in Version Undecided
  • Due Date Undecided
  • Votes 5
  • Private
Attached to Project: LEDE Project
Opened by camel - 25.05.2017
Last edited by Yousong Zhou - 01.09.2017

FS#804 - mt7621: kernel errors - rcu_sched detected stalls on CPUs/tasks

current trunk
hardware: zbt3526 mt7621
it give more and more often this kernel bugs ...
(i did not have that much for 2 months ago)
can it be related to newer kernel on TRUNK ?

Thu May 25 18:20:04 2017 user.notice root: Subject: [router.xxx.com] KERNEL error/warnings issue - 2017-05-25:18:20:01
Thu May 25 18:20:39 2017 kern.err kernel: [ 4797.640000] INFO: rcu_sched detected stalls on CPUs/tasks:
Thu May 25 18:20:39 2017 kern.err kernel: [ 4797.640000] 2-...: (0 ticks this GP) idle=dc4/0/0 softirq=370963/370963 fqs=0
Thu May 25 18:20:39 2017 kern.err kernel: [ 4797.650000] (detected by 1, t=6003 jiffies, g=119392, c=119391, q=19565)
Thu May 25 18:20:39 2017 kern.info kernel: [ 4797.650000] Task dump for CPU 2:
Thu May 25 18:20:39 2017 kern.info kernel: [ 4797.660000] swapper/2 R running 0 0 1 0×00100000 Thu May 25 18:20:39 2017 kern.warn kernel: [ 4797.660000] Stack : 00000000 00003a99 00000000 77de22c0 00000000 00000000 804df2a4 80490000
Thu May 25 18:20:39 2017 kern.warn kernel: [ 4797.660000] 8048c75c 00000001 00000000 8048c5e0 8048c724 80490000 00000000 800135e4
Thu May 25 18:20:39 2017 kern.warn kernel: [ 4797.660000] 00000000 814a37e0 8fc72000 8fc73ec0 80490000 8005ec74 1100fc03 00000002
Thu May 25 18:20:39 2017 kern.warn kernel: [ 4797.660000] 00000000 80490000 804df2a4 8005ec6c 80490000 8001b1a8 1100fc03 00000000
Thu May 25 18:20:39 2017 kern.warn kernel: [ 4797.660000] 00000004 8048c4a0 000000a0 8001b1b0 e8c7e2d3 3a8bf07f 2cfde824 eeff5ebf
Thu May 25 18:20:39 2017 kern.warn kernel: [ 4797.660000] ...
Thu May 25 18:20:39 2017 kern.warn kernel: [ 4797.700000] Call Trace:
Thu May 25 18:20:39 2017 kern.warn kernel: [ 4797.700000] [<8000be98>] __schedule+0×574/0×758 Thu May 25 18:20:39 2017 kern.warn kernel: [ 4797.710000] [<800135e4>] r4k_wait_irqoff+0×0/0×20 Thu May 25 18:20:39 2017 kern.warn kernel: [ 4797.710000]
Thu May 25 18:20:39 2017 kern.err kernel: [ 4797.710000] rcu_sched kthread starved for 6009 jiffies! g119392 c119391 f0x0 s3 →state=0×1

 


Closed by  Yousong Zhou
01.09.2017 11:07
Reason for closing:  Fixed
Cristian Morales Vega commented on 26.06.2017 13:54

It seems it has been happening for a while → http://lists.infradead.org/pipermail/lede-dev/2017-February/006325.html

I don't fully understand it, but I think it's related to these messages "cron.err crond[756]: time disparity of 1096 minutes detected". So it has real consequences, at the very least it can make crond sleep() for 18 hours without running any job.

Cristian Morales Vega commented on 26.06.2017 13:58

I have found this on a SK-WB8 (MT7621 too) with a slightly modified (not the kernel) 17.01.2.

Admin
john commented on 29.06.2017 17:25

i pushed a patch to my stging tree that might fix this issue
https://git.lede-project.org/?p=lede/blogic/staging.git;a=shortlog;h=refs/heads/lede-17.01

camel commented on 30.06.2017 00:45

let me know when the patch is in lede trunk on "ZBT-WG3526" image, and i will test.
cu Erwin

Baptiste Jonglez commented on 12.07.2017 09:26

You need to test the commit from john's staging tree before it gets merged into trunk.

By the way, ramips has been switched to linux 4.9, so it can also be worth testing the latest trunk image to see if it changes anything.

camel commented on 12.07.2017 11:06

well, if it would be in the trunk, we could test it ...eg: WG-3526 on which i can support for power testing :)
cu Erwin

lister-wrt commented on 13.07.2017 02:06

I'm having the same issue on Ubiquiti ERX. Can test patches but I don't know how to build it.

Baptiste Jonglez commented on 13.07.2017 21:11

I don't really like this, but here are all mt7621 images with lede-17.01 + john's patch (r3464+1-82b20d74cb): https://pub.polyno.me/lede-ramips-FS804/

Please only use these images for testing!

lister-wrt commented on 15.07.2017 09:12

Thanks Baptiste, I'll try it out. I have a USB-TTL in case it goes horribly wrong.

The only way I know of reliably reproducing this issue is with SQM (errors start after ~5m after install) and it's not in your build. Kernel 4.9.37 merged just after you built this so I won't be able to use the LEDE packages.

Is there another way to test?

dchard commented on 16.07.2017 21:49

Kernel 4.9.37 is also affected. I am testing John's patched build, and so far I was not able to reproduce this bug with hours of torturing the CPU. Previously it took only 5-10 minutes, so this is good news so far.

Jaap Buurman commented on 19.07.2017 10:54

This seems to be a duplicate or related to the following issue: https://bugs.lede-project.org/index.php?do=details&task_id=764

Unfortunately, during traffic shaping the Dir-860l still crashes with that patch applied. So it does not seem to be a complete fix. It does look like it takes longer for it to manifest, so I believe we're getting closer to the solution for our issues :)

camel commented on 19.07.2017 11:27

@Baptiste Jonglez
is it possible to rollout/commit your patch to TRUNK ?
as i want to test it on real env .. but therefore i need many extra kernel packages to install, too.
cu Erwin

camel commented on 22.07.2017 11:57

hm,, i tested current trunk ....(without traffic shapping packages installed and same result ...
Sat Jul 22 13:55:48 2017 kern.err kernel: [ 6282.000000] INFO: rcu_sched detected stalls on CPUs/tasks:
Sat Jul 22 13:55:48 2017 kern.err kernel: [ 6282.000000] 2-...: (0 ticks this GP) idle=550/0/0 softirq=447446/447446 fqs=1
Sat Jul 22 13:55:48 2017 kern.err kernel: [ 6282.010000] (detected by 3, t=6003 jiffies, g=152339, c=152338, q=631052)
Sat Jul 22 13:55:48 2017 kern.info kernel: [ 6282.010000] Task dump for CPU 2:
Sat Jul 22 13:55:48 2017 kern.info kernel: [ 6282.020000] swapper/2 R running task 0 0 1 0x00100000
Sat Jul 22 13:55:48 2017 kern.warn kernel: [ 6282.020000] Stack : 00000068 00000001 00000001 001bb9fe 00000000 778892c0 00000000 00000000
Sat Jul 22 13:55:48 2017 kern.warn kernel: [ 6282.030000] 804e148c 00000004 8047f934 804f0000 00000001 00000001 bfbf8000 00000000
Sat Jul 22 13:55:48 2017 kern.warn kernel: [ 6282.040000] 00000000 8000c1cc 00000000 00000000 8fc74000 8fc75f00 00000000 80062f44
Sat Jul 22 13:55:48 2017 kern.warn kernel: [ 6282.050000] 1100fc03 00000003 00000001 804e148c 804e148c 80062f3c 00000005 00000000
Sat Jul 22 13:55:48 2017 kern.warn kernel: [ 6282.060000] 00000000 00000001 00000005 00000000 00000000 00000001 00000001 803ec0b4
Sat Jul 22 13:55:48 2017 kern.warn kernel: [ 6282.070000] ...
Sat Jul 22 13:55:48 2017 kern.warn kernel: [ 6282.070000] Call Trace:
Sat Jul 22 13:55:48 2017 kern.warn kernel: [ 6282.070000] [<803e8408>] schedule+0x5d4/0x7a4
Sat Jul 22 13:55:48 2017 kern.warn kernel: [ 6282.080000] [<8000c1cc>] r4k_wait_irqoff+0x0/0x20
Sat Jul 22 13:55:48 2017 kern.err kernel: [ 6282.080000] rcu_sched kthread starved for 3035 jiffies! g152339 c152338 f0x0 RCU_GP_WAIT_FQS(3) →state=0x1
Sat Jul 22 13:55:48 2017 kern.info kernel: [ 6282.090000] rcu_sched S 0 7 2 0x00100000
Sat Jul 22 13:55:48 2017 kern.warn kernel: [ 6282.100000] Stack : 814a9bc0 8fc52340 8fc6be50 814a8320 8fc6be50 804e0000 00000001 8007b548
Sat Jul 22 13:55:48 2017 kern.warn kernel: [ 6282.100000] 814a8320 000914e6 814a8320 8fc6be50 804e0000 00000000 804f0000 804e0000
Sat Jul 22 13:55:48 2017 kern.warn kernel: [ 6282.110000] 804e96b0 803e8644 000914e6 804e0000 8fc6be50 814a8320 000914e6 803eafa8
Sat Jul 22 13:55:48 2017 kern.warn kernel: [ 6282.120000] 8fc6a000 8fc6be60 804e9580 804e96b0 814aa240 00000001 8e97f008 8e979408
Sat Jul 22 13:55:48 2017 kern.warn kernel: [ 6282.130000] 000914e6 8007ba44 8fc52340 44800002 804e9580 00000001 804e0000 00000000
Sat Jul 22 13:55:48 2017 kern.warn kernel: [ 6282.140000] ...
Sat Jul 22 13:55:48 2017 kern.warn kernel: [ 6282.140000] Call Trace:
Sat Jul 22 13:55:48 2017 kern.warn kernel: [ 6282.140000] [<803e8408>]
schedule+0x5d4/0x7a4
Sat Jul 22 13:55:48 2017 kern.warn kernel: [ 6282.150000] [<803e8644>] schedule+0x6c/0x84
Sat Jul 22 13:55:48 2017 kern.warn kernel: [ 6282.150000] [<803eafa8>] schedule_timeout+0x160/0x19c
Sat Jul 22 13:55:48 2017 kern.warn kernel: [ 6282.160000] [<80078f80>] rcu_gp_kthread+0x7f4/0x7fc
Sat Jul 22 13:55:48 2017 kern.warn kernel: [ 6282.160000] [<80044c68>] kthread+0xd8/0xec
Sat Jul 22 13:55:48 2017 kern.warn kernel: [ 6282.170000] [<8000a318>] ret_from_kernel_thread+0x14/0x1c

happening per day ~30 times ...
device: MT7621 - zbt3526

Admin
john commented on 22.07.2017 17:30

could you try if this still happens if remove
target/linux/ramips/patches-4.9/0005-MIPS-use-set_mode-to-enable-disable-the-cevt-r4k-irq.patch
target/linux/ramips/patches-4.9/0006-MIPS-ralink-add-cpu-frequency-scaling.patch

camel commented on 22.07.2017 17:47

Sorry, can't build own image.
If available on trunk to DL, I will test

Pierre Parent commented on 23.07.2017 11:41

@John

I have the same problem.

I will try to compile without those patches tomorrow, also for me the bug is not as easily replicable as camel say, at least without traffic shaping and without mt7603e driver it's rare.

Also if you want a ZBT-WG3526, to be able to test yourself , I can send you one for free.

camel commented on 23.07.2017 15:31

For me it is clear related to wlan 2,4GHZ
as currently I'm copying a lot of data over 2,4 ghz, and I have rcu-sched kennel errors all 5minutes

Pierre Parent commented on 24.07.2017 07:23

@camel:

Not it's probably not, because the problem happens without 2,4Ghz driver running.

Though 2.4Ghz driver can make the problem happen more often, and the driver mt76 itself has some specific issues independent from that problem.

Traffic shaping makes it happen more often even with 2.4Ghz driver disabled.

Pierre Parent commented on 24.07.2017 08:02

I compiled the version without patch
target/linux/ramips/patches-4.9/0005-MIPS-use-set_mode-to-enable-disable-the-cevt-r4k-irq.patch
target/linux/ramips/patches-4.9/0006-MIPS-ralink-add-cpu-frequency-scaling.patch

You can download it here:

https://www.own-mailbox.com/lede/lede-ramips-mt7621-zbt-wg3526-16M-squashfs-sysupgrade.bin

Please can you test, I will test on my side when I have time

Edit: I have updated firmware on my server with few more packages including kmod-mt76 and kmod-sched at 8:40 GMT md5sum: 9ac127a3d0bf49a8d452e51b2ff9b741

camel commented on 24.07.2017 08:09

I can't test, as I would med more other packages .
I can only test trunk version, on which I can install my other mandatory packages

Pierre Parent commented on 24.07.2017 08:41

I have updated firmware on my server with few more packages including kmod-mt76 and kmod-sched at 8:40 GMT md5sum: 9ac127a3d0bf49a8d452e51b2ff9b741

What packages would you need that you cannot install with opkg? (Only kernel related packages cannot be installed with opkg)

camel commented on 24.07.2017 12:14

is too much, and i don'T know exactly which packackes are then needed to reflect kernel builds ...

but if interested .. this is my list of packages what I'm installing ...

#disk & SD related stuff
opkg install block-mount # --force-reinstall
opkg install kmod-scsi-core kmod-usb-storage #--force-reinstall
opkg install kmod-fs-ext4 kmod-fs-vfat 
opkg install kmod-nls-utf8 kmod-nls-cp437 kmod-nls-iso8859-1 #--force-reinstall
opkg install kmod-fs-nfs nfs-utils #--force-reinstall
opkg install kmod-fs-ext4 kmod-fs-vfat kmod-nls-utf8 kmod-nls-base kmod-nls-cp437 kmod-nls-iso8859-1 cfdisk e2fsprogs #--force-reinstall
opkg install kmod-fs-f2fs libf2fs f2fs-tools f2fsck mkf2fs
opkg install fdisk  #--force-reinstall
opkg install rsync

#WEBIFACE:
opkg install luci #--force-reinstall
opkg install luci-ssl-openssl

#luci extra stuff:
opkg install luci-app-commands #--force-reinstall
opkg install luci-app-ddns #--force-reinstall
opkg install luci-app-diag-core #--force-reinstall
opkg install luci-app-firewall #--force-reinstall
opkg install luci-app-ntpc #--force-reinstall
opkg install luci-app-uhttpd #--force-reinstall

#3g/4g + luci (https://wiki.openwrt.org/doc/recipes/3gdongle)
opkg install chat #--force-reinstall
opkg install comgt #--force-reinstall
opkg install luci-proto-3g #--force-reinstall
opkg install kmod-usb-serial #--force-reinstall
opkg install kmod-usb-serial-option #--force-reinstall
opkg install kmod-usb-serial-wwan #--force-reinstall #(mostly added now in current release or trunk)

#3ginfo package
opkg install http://dl.eko.one.pl/chaos_calmer/ar71xx/packages/3ginfo-text_20170524_all.ipk
#opkg install http://dl.eko.one.pl/chaos_calmer/ar71xx/packages/3ginfo__20170524_all.ipk --force-checksum --force-reinstall

#pptp + pptpd =>
#=======
opkg install kmod-crypto-manager kmod-crypto-pcompress kmod-crypto-ecb
opkg install  kmod-crypto-null kmod-crypto-aead kmod-iptunnel kmod-gre resolveip kmod-pptp kmod-crypto-sha1
opkg install ppp-mod-pptp kmod-nf-nathelper-extra
#is for routing vpn client to vpn client and lan to vpn-client
opkg install kmod-ipt-nat-extra
opkg install luci-proto-ppp

#VPN PPTP server:
opkg install pptpd kmod-mppe

#OPENVPN: (ca. 1MB space needed)
opkg install openvpn-mbedtls luci-app-openvpn
opkg install nano libustream-openssl ca-bundle ca-certificates

# logging + rotate
opkg install logrotate --force-reinstall

#monitoring ....
opkg install luci-app-statistics --force-reinstall
opkg install collectd-mod-conntrack --force-reinstall
opkg install collectd-mod-cpu --force-reinstall
opkg install collectd-mod-df --force-reinstall
opkg install collectd-mod-disk --force-reinstall
opkg install collectd-mod-exec --force-reinstall
opkg install collectd-mod-interface --force-reinstall
opkg install collectd-mod-iptables --force-reinstall
opkg install collectd-mod-iwinfo --force-reinstall
opkg install collectd-mod-load --force-reinstall
opkg install collectd-mod-memory --force-reinstall
opkg install collectd-mod-ping --force-reinstall
opkg install collectd-mod-processes --force-reinstall
opkg install collectd-mod-rrdtool --force-reinstall
opkg install collectd-mod-uptime --force-reinstall

#monitoring data traffic (optional) ~
opkg install luci-app-vnstat --force-reinstall

#ftp server:
opkg install  --force-maintainer --force-reinstall --force-depends  --force-overwrite vsftpd

#traffic shaping:
opkg install luci-app-sqm sqm-scripts kmod-sched-cake --force-reinstall

#loadbalancing:
opkg install luci-app-mwan3 --force-reinstall

#HIGH AVAILIBILITY
opkg install keepalived conntrackd --force-reinstall

#email + ssl (~1MB) - also see: https://wiki.openwrt.org/doc/howto/smtp.client
opkg install libopenssl ssmtp --force-reinstall

#smstools + web (~2,3MB needed)
opkg install smstools3 --force-reinstall

#to USB ...
opkg  --add-dest opt:/opt install php7-cgi php7-mod-session php7-mod-json php7-mod-curl --force-reinstall
ln -s /opt/usr/bin/php-cgi /usr/bin/php-cgi
ln -s /opt/usr/lib/libstdc++.so.6 /usr/lib/libstdc++.so.6

#optional file systems tools
opkg install lsof fuse-utils --force-reinstall

#extra apps
opkg install libpcap --force-reinstall
opkg install terminfo --force-reinstall
opkg install rsync --force-reinstall
opkg install sudo --force-reinstall
opkg install shadow-useradd shadow-su --force-reinstall
opkg install coreutils-date --force-reinstall
opkg install glib2 --force-reinstall #libgcc
opkg install mc --force-reinstall

#debugging GSM modem:
opkg  --add-dest opt:/opt install picocom --force-reinstall

# others
opkg --add-dest opt:/opt install screen --force-reinstall
opkg --add-dest opt:/opt install htop --force-reinstall
opkg --add-dest opt:/opt install procps-ng-watch --force-reinstall
opkg --add-dest opt:/opt install nmap --force-reinstall
opkg --add-dest opt:/opt install tcpdump-mini --force-reinstall
opkg --add-dest opt:/opt install sysstat --force-reinstall
opkg --add-dest opt:/opt install confuse --force-reinstall


#for speedtest: (in 3g package given or also on USB stick - speedtest dir)
opkg --add-dest opt:/opt install curl --force-reinstall
opkg --add-dest opt:/opt install bc --force-reinstall
opkg --add-dest opt:/opt install python # python-base --force-reinstall
opkg --add-dest opt:/opt install --force-reinstall --force-depends python python-base python-codecs python-compiler python-ctypes python-db python-decimal python-distutils python-email python-gdbm python-light python-logging python-multiprocessing python-ncurses python-openssl python-pydoc python-sqlite3 python-unittest python-xml

#console tools -  measurement tools
opkg --add-dest opt:/opt install iperf --force-reinstall
opkg --add-dest opt:/opt install bmon libnl-route --force-reinstall
opkg --add-dest opt:/opt install iftop --force-reinstall
opkg --add-dest opt:/opt install iptraf-ng --force-reinstall
camel commented on 24.07.2017 12:16

well, if wanted i can try to install .... and let you know whcih packages i would missing on kernel related stuff ...

which link for your build can i use ?

i tried it, but it is too much what would be missing to test it longer ...

no luci packages, no modem driver, etc ...

i need to wait till it is in trunk.

i guess, it can not be more worst as it is for now in current trunk ...
and would hope that your patches will be commited soon .. thx
then i will TEST ASAP

Zhong Jianxin commented on 24.07.2017 12:52

It's possible to force building the same kernel version as upstream snapshot, e.g:

  $ make clean    
  $ make LINUX_VERMAGIC=7f0de30d5b73958cb146494d8e5b2ef4

As long as you use the same code base and same kernel config, kmod from upstream snapshot should work fine.

Pierre Parent commented on 24.07.2017 13:06

I've started testing with qos enabled but mt76 (2.4ghz driver) disabled, with my image, for now it seems that I don't see any RCU_sched warning anymore but I would need confirmation since for me it was always very random, and not easily reproducible.

I guess if there is none to test, I don't know if anything will get to trunk soon. Especially Since this hack is about removing patches impacting all MIPS images. But john should know better than I do.

Here is the latest image:
https://www.own-mailbox.com/lede/lede-ramips-mt7621-zbt-wg3526-16M-squashfs-sysupgrade.bin

The image builder:
https://www.own-mailbox.com/lede/lede-imagebuilder-ramips-mt7621.Linux-x86_64.tar.xz

The Sdk:
https://www.own-mailbox.com/lede/lede-sdk-ramips-mt7621_gcc-5.4.0_musl.Linux-x86_64.tar.xz

Pierre Parent commented on 24.07.2017 13:09

@Zhong Jianxin: will it work even with a modified kernel (since we change patches used in upstream)?

camel commented on 24.07.2017 13:11

@pierre:
Pls can you build with same like snapshot ?
As described above...

It's possible to force building the same kernel version as upstream snapshot, e.g:

$ make clean    
$ make LINUX_VERMAGIC=7f0de30d5b73958cb146494d8e5b2ef4

As long as you use the same code base and same kernel config, kmod from upstream snapshot should work fine

Pierre Parent commented on 24.07.2017 13:28

I've updated the image+sdk+image-builder on my server compiled with the above command.

The md5sum of the image is 3298cb86e8ff7737fcad8bc4065914ec.

Please test, if you can install your packages with it.

camel commented on 24.07.2017 13:52

ok, first try ..
if i try to install the:
*opkg install kmod-fs-nfs nfs-utils* it was crashing, image bricked and going into a boot loop.
now set it back to chinese FW .. and now installing it again - and trying only to use what is really needed (ext4 + f2fs)

rsync package seems to be completely missing now on the TRUNK snapshot .. maybe compile failed on some new packages ...
( :( → needs to be reported to devels anyhow ..)

i let you know as soon as i get it installed


maybe some small issues .. not sure, if i can mount all OK ...
Mon Jul 24 13:46:48 2017 daemon.notice squid[1531]: Logfile: opening log stdio:/var/logs/netdb.state
Mon Jul 24 13:46:48 2017 daemon.notice squid[1531]: ERROR: logfileOpen stdio:/var/logs/netdb.state: (2) No such file or directory
Mon Jul 24 13:46:48 2017 daemon.notice squid[1531]: logfileOpen: stdio:/var/logs/netdb.state: couldn't open!
Mon Jul 24 13:46:48 2017 daemon.notice squid[1531]: netdbSaveState: stdio:/var/logs/netdb.state: (2) No such file or directory
Mon Jul 24 13:46:57 2017 daemon.info dnsmasq[2111]: read /etc/hosts - 4 addresses
Mon Jul 24 13:46:57 2017 daemon.info dnsmasq[2111]: read /tmp/hosts/dhcp.cfg02411c - 2 addresses
Mon Jul 24 13:46:57 2017 daemon.info dnsmasq-dhcp[2111]: read /etc/ethers - 0 addresses
Mon Jul 24 13:49:15 2017 kern.warn kernel: [ 267.690000] f2fs: Unknown symbol posix_acl_chmod (err 0)

camel commented on 24.07.2017 13:56

hmm :( ..
f2fs can'T be used ... which i need as there are all the logs analysing scripts were written to it :(
can you try if you can include that into your image ?
on mounting it give me that error:
Mon Jul 24 13:55:03 2017 kern.warn kernel: [ 615.690000] f2fs: Unknown symbol posix_acl_chmod (err 0)
Mon Jul 24 13:55:03 2017 daemon.err modprobe: 1 module could not be probed
Mon Jul 24 13:55:03 2017 daemon.err modprobe: - f2fs
Mon Jul 24 13:55:04 2017 kern.warn kernel: [ 615.790000] f2fs: Unknown symbol posix_acl_chmod (err 0)
Mon Jul 24 13:55:04 2017 daemon.err modprobe: 1 module could not be probed
Mon Jul 24 13:55:04 2017 daemon.err modprobe: - f2fs
Mon Jul 24 13:55:04 2017 kern.warn kernel: [ 615.910000] f2fs: Unknown symbol posix_acl_chmod (err 0)
Mon Jul 24 13:55:04 2017 daemon.err modprobe: 1 module could not be probed
Mon Jul 24 13:55:04 2017 daemon.err modprobe: - f2fs
Mon Jul 24 13:55:04 2017 kern.warn kernel: [ 616.010000] f2fs: Unknown symbol posix_acl_chmod (err 0)
Mon Jul 24 13:55:04 2017 daemon.err modprobe: 1 module could not be probed
Mon Jul 24 13:55:04 2017 daemon.err modprobe: - f2fs

Pierre Parent commented on 24.07.2017 14:06

probably the kernel modules are not compatible because they were not built against the exact same kernel so Zhong Jianxin's technique is not goood.

Also many packages are currently missing from trunk but I guess they will come back soon.

camel commented on 24.07.2017 14:18

hmm, tried it again ....

yes, seems to be more packages missing or not fully correct with build options ...

root@LEDE:~# opkg install ppp-mod-pptp kmod-nf-nathelper-extra
Installing ppp-mod-pptp (2.4.7-12) to root...
Downloading http://downloads.lede-project.org/snapshots/packages/mipsel_24kc/base/ppp-mod-pptp_2.4.7-12_mipsel_24kc.ipk Installing kmod-mppe (4.9.37-1) to root...
Downloading http://downloads.lede-project.org/snapshots/targets/ramips/mt7621/packages/kmod-mppe_4.9.37-1_mipsel_24kc.ipk Installing kmod-nf-nathelper-extra (4.9.37-1) to root...
Downloading http://downloads.lede-project.org/snapshots/targets/ramips/mt7621/packages/kmod-nf-nathelper-extra_4.9.37-1_mipsel_24kc.ipk Installing kmod-lib-textsearch (4.9.37-1) to root...
Downloading http://downloads.lede-project.org/snapshots/targets/ramips/mt7621/packages/kmod-lib-textsearch_4.9.37-1_mipsel_24kc.ipk Configuring kmod-lib-textsearch.
Configuring kmod-mppe.
Configuring ppp-mod-pptp.
Configuring kmod-nf-nathelper-extra.
2 modules could not be probed
- nf_conntrack_pptp
- nf_conntrack_proto_gre
1 module could not be probed
- nf_conntrack_proto_gre
4 modules could not be probed
- nf_conntrack_pptp
- nf_conntrack_proto_gre
- nf_nat_pptp
- nf_nat_proto_gre
1 module could not be probed
- nf_nat_proto_gre

then i saw in logs:

Mon Jul 24 14:06:52 2017 kern.warn kernel: [ 1324.590000] ip_tunnel: Unknown symbol dst_cache_set_ip4 (err 0)
Mon Jul 24 14:06:52 2017 kern.warn kernel: [ 1324.600000] ip_tunnel: Unknown symbol dst_cache_destroy (err 0)
Mon Jul 24 14:06:52 2017 kern.warn kernel: [ 1324.600000] ip_tunnel: Unknown symbol dst_cache_init (err 0)
Mon Jul 24 14:06:52 2017 kern.warn kernel: [ 1324.610000] ip_tunnel: Unknown symbol dst_cache_get_ip4 (err 0)
Mon Jul 24 14:06:52 2017 kern.info kernel: [ 1324.730000] gre: GRE over IPv4 demultiplexor driver
Mon Jul 24 14:06:53 2017 kern.warn kernel: [ 1324.740000] ip_gre: Unknown symbol ip_tunnel_rcv (err 0)
Mon Jul 24 14:06:53 2017 kern.warn kernel: [ 1324.750000] ip_gre: Unknown symbol ip_tunnel_change_mtu (err 0)
Mon Jul 24 14:06:53 2017 kern.warn kernel: [ 1324.750000] ip_gre: Unknown symbol ip_tunnel_encap_setup (err 0)
Mon Jul 24 14:06:53 2017 kern.warn kernel: [ 1324.760000] ip_gre: Unknown symbol dst_cache_set_ip4 (err 0)
Mon Jul 24 14:06:53 2017 kern.warn kernel: [ 1324.760000] ip_gre: Unknown symbol ip_tunnel_init_net (err 0)
Mon Jul 24 14:06:53 2017 kern.warn kernel: [ 1324.770000] ip_gre: Unknown symbol ip_tunnel_changelink (err 0)
Mon Jul 24 14:06:53 2017 kern.warn kernel: [ 1324.780000] ip_gre: Unknown symbol ip_tunnel_ioctl (err 0)
Mon Jul 24 14:06:53 2017 kern.warn kernel: [ 1324.780000] ip_gre: Unknown symbol ip_tunnel_get_link_net (err 0)
Mon Jul 24 14:06:53 2017 kern.warn kernel: [ 1324.790000] ip_gre: Unknown symbol ip_tunnel_init (err 0)
Mon Jul 24 14:06:53 2017 kern.warn kernel: [ 1324.790000] ip_gre: Unknown symbol ip_tunnel_delete_net (err 0)
Mon Jul 24 14:06:53 2017 kern.warn kernel: [ 1324.800000] ip_gre: Unknown symbol ip_tunnel_uninit (err 0)
Mon Jul 24 14:06:53 2017 kern.warn kernel: [ 1324.810000] ip_gre: Unknown symbol ip_tunnel_newlink (err 0)
Mon Jul 24 14:06:53 2017 kern.warn kernel: [ 1324.810000] ip_gre: Unknown symbol ip_tunnel_dellink (err 0)
Mon Jul 24 14:06:53 2017 kern.warn kernel: [ 1324.820000] ip_gre: Unknown symbol ip_tunnel_setup (err 0)
Mon Jul 24 14:06:53 2017 kern.warn kernel: [ 1324.820000] ip_gre: Unknown symbol ip_tunnel_xmit (err 0)
Mon Jul 24 14:06:53 2017 kern.warn kernel: [ 1324.830000] ip_gre: Unknown symbol dst_cache_get_ip4 (err 0)
Mon Jul 24 14:06:53 2017 kern.warn kernel: [ 1324.830000] ip_gre: Unknown symbol __ip_tunnel_change_mtu (err 0)
Mon Jul 24 14:06:53 2017 kern.warn kernel: [ 1324.840000] ip_gre: Unknown symbol ip_tunnel_lookup (err 0)
Mon Jul 24 14:06:53 2017 kern.warn kernel: [ 1324.850000] ip_gre: Unknown symbol ip_tunnel_get_iflink (err 0)
Mon Jul 24 14:06:53 2017 kern.info kernel: [ 1324.990000] PPTP driver version 0.8.5
Mon Jul 24 14:07:00 2017 kern.info kernel: [ 1332.670000] PPP MPPE Compression module registered
Mon Jul 24 14:07:01 2017 kern.warn kernel: [ 1332.950000] nf_conntrack_proto_gre: Unknown symbol nf_ct_port_nlattr_tuple_size (err 0)
Mon Jul 24 14:07:01 2017 kern.warn kernel: [ 1332.950000] nf_conntrack_proto_gre: Unknown symbol nf_ct_port_nla_policy (err 0)
Mon Jul 24 14:07:01 2017 kern.warn kernel: [ 1332.960000] nf_conntrack_proto_gre: Unknown symbol nf_ct_port_tuple_to_nlattr (err 0)
Mon Jul 24 14:07:01 2017 kern.warn kernel: [ 1332.970000] nf_conntrack_proto_gre: Unknown symbol nf_ct_port_nlattr_to_tuple (err 0)
Mon Jul 24 14:07:01 2017 kern.warn kernel: [ 1333.070000] nf_conntrack_proto_gre: Unknown symbol nf_ct_port_nlattr_tuple_size (err 0)
Mon Jul 24 14:07:01 2017 kern.warn kernel: [ 1333.080000] nf_conntrack_proto_gre: Unknown symbol nf_ct_port_nla_policy (err 0)
Mon Jul 24 14:07:01 2017 kern.warn kernel: [ 1333.090000] nf_conntrack_proto_gre: Unknown symbol nf_ct_port_tuple_to_nlattr (err 0)
Mon Jul 24 14:07:01 2017 kern.warn kernel: [ 1333.090000] nf_conntrack_proto_gre: Unknown symbol nf_ct_port_nlattr_to_tuple (err 0)
Mon Jul 24 14:07:01 2017 kern.warn kernel: [ 1333.380000] nf_conntrack_proto_gre: Unknown symbol nf_ct_port_nlattr_tuple_size (err 0)
Mon Jul 24 14:07:01 2017 kern.warn kernel: [ 1333.390000] nf_conntrack_proto_gre: Unknown symbol nf_ct_port_nla_policy (err 0)
Mon Jul 24 14:07:01 2017 kern.warn kernel: [ 1333.400000] nf_conntrack_proto_gre: Unknown symbol nf_ct_port_tuple_to_nlattr (err 0)
Mon Jul 24 14:07:01 2017 kern.warn kernel: [ 1333.400000] nf_conntrack_proto_gre: Unknown symbol nf_ct_port_nlattr_to_tuple (err 0)
Mon Jul 24 14:07:01 2017 kern.warn kernel: [ 1333.460000] nf_nat_proto_gre: Unknown symbol nf_nat_l4proto_nlattr_to_range (err 0)
Mon Jul 24 14:07:01 2017 kern.warn kernel: [ 1333.530000] nf_nat_proto_gre: Unknown symbol nf_nat_l4proto_nlattr_to_range (err 0)

then i proceed with other packages (monitoring) .. and then i got a crash ...
Mon Jul 24 14:10:45 2017 kern.alert kernel: [ 1557.110000] CPU 2 Unable to handle kernel paging request at virtual address 00000000, epc == 8efd1970, ra == 8efd1970
Mon Jul 24 14:10:45 2017 kern.warn kernel: [ 1557.120000] Oops[#1]:
Mon Jul 24 14:10:45 2017 kern.warn kernel: [ 1557.120000] CPU: 2 PID: 9671 Comm: insmod Not tainted 4.9.37 #0
Mon Jul 24 14:10:45 2017 kern.warn kernel: [ 1557.130000] task: 8fe8f580 task.stack: 8eafa000
Mon Jul 24 14:10:45 2017 kern.warn kernel: [ 1557.130000] $ 0 : 00000000 00000069 00040925 00040925
Mon Jul 24 14:10:45 2017 kern.warn kernel: [ 1557.140000] $ 4 : 8eafbc74 00000000 00000000 27c82c2a
Mon Jul 24 14:10:45 2017 kern.warn kernel: [ 1557.140000] $ 8 : 00000000 801e2050 00000000 80556070
Mon Jul 24 14:10:45 2017 kern.warn kernel: [ 1557.150000] $12 : 8eafbbe0 00000000 00000000 124dab96
Mon Jul 24 14:10:45 2017 kern.warn kernel: [ 1557.150000] $16 : 8fec0800 00000000 8efd19ac 00000001
Mon Jul 24 14:10:45 2017 kern.warn kernel: [ 1557.160000] $20 : 00000000 8efd18a4 8efd0000 024000c0
Mon Jul 24 14:10:45 2017 kern.warn kernel: [ 1557.160000] $24 : 00000002 df6052aa
Mon Jul 24 14:10:45 2017 kern.warn kernel: [ 1557.170000] $28 : 8eafa000 8eafbc90 80091f7c 8efd1970
Mon Jul 24 14:10:45 2017 kern.warn kernel: [ 1557.170000] Hi : 00000000
Mon Jul 24 14:10:45 2017 kern.warn kernel: [ 1557.180000] Lo : 00000000
Mon Jul 24 14:10:45 2017 kern.warn kernel: [ 1557.180000] epc : 8efd1970 init_module+0xbca970/0xbca9a0 [ifb]
Mon Jul 24 14:10:45 2017 kern.warn kernel: [ 1557.180000] ra : 8efd1970 init_module+0xbca970/0xbca9a0 [ifb]
Mon Jul 24 14:10:45 2017 kern.warn kernel: [ 1557.190000] Status: 1100fc03 KERNEL EXL IE Mon Jul 24 14:10:45 2017 kern.warn kernel: [ 1557.190000] Cause : 40800008 (ExcCode 02)
Mon Jul 24 14:10:45 2017 kern.warn kernel: [ 1557.200000] BadVA : 00000000
Mon Jul 24 14:10:45 2017 kern.warn kernel: [ 1557.200000] PrId : 0001992f (MIPS 1004Kc)
Mon Jul 24 14:10:45 2017 kern.warn kernel: [ 1557.210000] Modules linked in: ifb(+) xt_NETMAP nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_irc nf_nat_h323 nf_nat_amanda nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_irc nf_conntrack_h323 nf_conntrack_broadcast nf_conntrack_amanda ppp_mppe ts_kmp ts_fsm ts_bm sha1_generic pptp gre ecb option usb_wwan usbserial crc32_generic vfat fat nls_utf8 nls_iso8859_1 nls_cp437 pppoe ppp_async ath9k rtl8821ae rtl8192se rtl8192de rtl8192cu rtl8192c_

and then boot loop ... :(

camel commented on 24.07.2017 14:46

well, i think, i can try without the pptp stuff , but the f2fs would be really helpful as there i have many things located (logs, rrd, etc ..)

Mon Jul 24 14:44:14 2017 kern.warn kernel: [ 1320.910000] f2fs: Unknown symbol posix_acl_chmod (err 0)

not sure, if you can try to add "F2FS" to built easily ?
using this packages:
#f2fs - for flash mem USB stick
opkg install kmod-fs-f2fs libf2fs f2fs-tools f2fsck mkf2fs

cu

Pierre Parent commented on 24.07.2017 15:41

After I continued testing I finally got a message:

[  328.780000] INFO: rcu_sched detected stalls on CPUs/tasks:
[  328.780000] 	1-...: (1 GPs behind) idle=15a/0/0 softirq=17882/17884 fqs=0 
[  328.780000] 	3-...: (0 ticks this GP) idle=39c/0/0 softirq=19473/19473 fqs=0 
[  328.780000] 	(detected by 0, t=6002 jiffies, g=6144, c=6143, q=27828)
[  328.780000] Task dump for CPU 1:
[  328.780000] swapper/1       R  running task        0     0      1 0x00100000
[  328.780000] Stack : 8fc73fe0 0000fc00 00000000 00001400 00000000 00000009 00000000 0000024e
[  328.780000]         804bf48c 00000002 8045f424 804d0000 00000001 00000001 bfbf8000 00000001
[  328.780000]         00000000 8000c1cc 00000000 00000000 8fc72000 8fc73f00 00000000 80062ea4
[  328.780000]         1100fc03 00000003 00000001 804d0000 804bf48c 80062e9c 00000000 00000001
[  328.780000]         00000001 00000001 00000000 00000001 00000001 00000001 00000001 9fc0046c
[  328.780000]         ...
[  328.780000] Call Trace:
[  328.780000] [<803ca7b8>] __schedule+0x5d4/0x7a4
[  328.780000] [<8000c1cc>] r4k_wait_irqoff+0x0/0x20
[  328.780000] Task dump for CPU 3:
[  328.780000] swapper/3       R  running task        0     0      1 0x00100000
[  328.780000] Stack : 00000068 00000001 00000001 001fb940 00000000 77ddc2c0 00000000 00000000
[  328.780000]         804bf48c 00000008 8045f424 804d0000 00000001 00000001 bfbf8000 00000003
[  328.780000]         00000000 8000c1cc 00000000 00000000 8fc76000 8fc77f00 00000000 80062ea4
[  328.780000]         1100fc03 00000003 00000001 804bf48c 804bf48c 80062e9c 00000001 00000001
[  328.780000]         00000001 00000001 00000001 00000001 00000001 00000001 00000001 9fc0046c
[  328.780000]         ...
[  328.780000] Call Trace:
[  328.780000] [<803ca7b8>] __schedule+0x5d4/0x7a4
[  328.780000] [<8000c1cc>] r4k_wait_irqoff+0x0/0x20
[  328.780000] rcu_sched kthread starved for 6002 jiffies! g6144 c6143 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1
[  328.780000] rcu_sched       S    0     7      2 0x00100000
[  328.780000] Stack : 814b5bc0 8fc52340 8fc6be50 814b4320 8fc6be50 00000000 00000000 8007b4a8
[  328.780000]         814b4320 fffff3cd 814b4320 8fc6be50 804c0000 00000001 804c0000 804c0000
[  328.780000]         804c75b0 803ca9f4 fffff3cd 804c0000 8fc6be50 814b4320 fffff3cd 803cd358
[  328.780000]         804bf5f4 00000000 804c0000 804c0000 814b6240 00000001 00000000 814b43b0
[  328.780000]         fffff3cd 8007b9a4 8fc52340 03800003 804c7480 00000001 804c0000 00000000
[  328.780000]         ...
[  328.780000] Call Trace:
[  328.780000] [<803ca7b8>] __schedule+0x5d4/0x7a4
[  328.780000] [<803ca9f4>] schedule+0x6c/0x84
[  328.780000] [<803cd358>] schedule_timeout+0x160/0x19c
[  328.780000] [<80078ee0>] rcu_gp_kthread+0x7f4/0x7fc
[  328.780000] [<80044bc8>] kthread+0xd8/0xec
[  328.780000] [<8000a318>] ret_from_kernel_thread+0x14/0x1c

After that typing "sleep 1" in the console would hang forever

Any other idea?

camel commented on 24.07.2017 15:52

wau, i tested now again your version, and did a lot of load on network
i never have seen the 5GHZ so fast like now ...( no clue why) ...
DL was with real 26MByte/sec and upload 20MB/sec.
additional, i didn't got any more RCU_SCHED errors.
it is now for me looking really very very good.

and i can simulate it easily .... on current trunk:
produce a lot of network traffic (LAN, WLAN2,4GHZ/5GHZ) ....
and copy >2GB of data, then you get it for sure ....

and your version ...
copied now 15GBwithout any alarm or logs entry !!!!
GREAT
so, i can only tell you, it is much more stable as before. for 100% sure !!!

and praying to get it to the latest trunk commited to test more in detail and let it running.

currently i didn't install all the pptp for client and server support ... and also the F2FS missing.
additional, i have seen that the package: keepalaive was also using a package:

Mon Jul 24 17:08:00 2017 kern.warn kernel: [ 17.920000] fuse: Unknown symbol forget_all_cached_acls (err 0)
Mon Jul 24 17:08:00 2017 kern.warn kernel: [ 17.920000] fuse: Unknown symbol posix_acl_default_xattr_handler (err 0)
Mon Jul 24 17:08:00 2017 kern.warn kernel: [ 17.930000] fuse: Unknown symbol posix_acl_from_xattr (err 0)
Mon Jul 24 17:08:00 2017 kern.warn kernel: [ 17.930000] fuse: Unknown symbol posix_acl_access_xattr_handler (err 0)
Mon Jul 24 17:08:00 2017 kern.warn kernel: [ 17.940000] fuse: Unknown symbol posix_acl_to_xattr (err 0)
Mon Jul 24 17:08:00 2017 kern.warn kernel: [ 18.000000] nf_conntrack_netlink: Unknown symbol nf_conntrack_register_notifier (err 0)
Mon Jul 24 17:08:00 2017 kern.warn kernel: [ 18.000000] nf_conntrack_netlink: Unknown symbol nf_conntrack_unregister_notifier (err 0)
Mon Jul 24 17:08:00 2017 kern.warn kernel: [ 18.010000] nf_conntrack_netlink: Unknown symbol nf_ct_expect_unregister_notifier (err 0)
Mon Jul 24 17:08:00 2017 kern.warn kernel: [ 18.020000] nf_conntrack_netlink: Unknown symbol nf_ct_expect_register_notifier (err 0)
Mon Jul 24 17:08:00 2017 kern.warn kernel: [ 18.030000] nf_conntrack_netlink: Unknown symbol nf_conntrack_eventmask_report (err 0)

but that is only related to your build i guess ....

let me know whom i need to kiss the ass to get it commited on current trunk snapshot :)?
then i will do more testing ...

Pierre Parent commented on 24.07.2017 16:02

I've update the image: go back to normal kernel-magic and add few kernel modules.

It may be more stable but it does not solve completely rcu_sched problems, as I still got a very problematic situation where my router is uncontrollable and only reboot by unplugging is possible.

Also I'm pretty sure that with openwrt 15.05.1 and kernel 3.18.23 there is no such rcu problem at all.

let me know whom i need to kiss the ass to get it commited on current trunk snapshot :)?
Ps: I highly doubt it will as such since it would impact all MIPS devices (and there are many!) and it does not even fully solve our problem.
camel commented on 24.07.2017 16:29

?
what do you mean ?
i can install current lede trunk ?
commit done ?
or do you mean, i should try to reinstall yours again ? + then trying with f2fs and pptp client & server related packages should be OK with your new one ?
what did you changed ?

camel commented on 24.07.2017 16:31

to your comment:
Also I'm pretty sure that with openwrt 15.05.1 and kernel 3.18.23 there is no such rcu problem at all.

Yes, you are right, i tested the old wrt kernel 3.x .. and is not having such kind of problems.
also tested from manufacturer the image, this is the stablest one. but useless as totally out of date.

camel commented on 24.07.2017 16:57

tried now yours .. puh, that is now really a big image ...
anyhow ...

but still i get:

Downloading http://downloads.lede-project.org/snapshots/targets/ramips/mt7621/packages/kmod-crypto-sha1_4.9.37-1_mipsel_24kc.ipk Configuring resolveip.
Collected errors:
* satisfy_dependencies_for: Cannot satisfy the following dependencies for kmod-iptunnel:
* kernel (= 4.9.37-1-7f0de30d5b73958cb146494d8e5b2ef4) *
* opkg_install_cmd: Cannot install package kmod-iptunnel.
* satisfy_dependencies_for: Cannot satisfy the following dependencies for kmod-gre:
* kernel (= 4.9.37-1-7f0de30d5b73958cb146494d8e5b2ef4) *
* opkg_install_cmd: Cannot install package kmod-gre.
* satisfy_dependencies_for: Cannot satisfy the following dependencies for kmod-pptp:
* kernel (= 4.9.37-1-7f0de30d5b73958cb146494d8e5b2ef4) *
* opkg_install_cmd: Cannot install package kmod-pptp.
* satisfy_dependencies_for: Cannot satisfy the following dependencies for kmod-crypto-sha1:
* kernel (= 4.9.37-1-7f0de30d5b73958cb146494d8e5b2ef4) *
* opkg_install_cmd: Cannot install package kmod-crypto-sha1.
root@LEDE:~#
root@LEDE:~# opkg install ppp-mod-pptp kmod-nf-nathelper-extra
Installing ppp-mod-pptp (2.4.7-12) to root...
Downloading http://downloads.lede-project.org/snapshots/packages/mipsel_24kc/base/ppp-mod-pptp_2.4.7-12_mipsel_24kc.ipk Installing kmod-nf-nathelper-extra (4.9.37-1) to root...
Downloading http://downloads.lede-project.org/snapshots/targets/ramips/mt7621/packages/kmod-nf-nathelper-extra_4.9.37-1_mipsel_24kc.ipk Collected errors:
* satisfy_dependencies_for: Cannot satisfy the following dependencies for ppp-mod-pptp:
* kernel (= 4.9.37-1-7f0de30d5b73958cb146494d8e5b2ef4) * kernel (= 4.9.37-1-7f0de30d5b73958cb146494d8e5b2ef4) * kernel (= 4.9.37-1-7f0de30d5b73958cb146494d8e5b2ef4) * kernel (= 4.9.37-1-7f0de30d5b73958cb146494d8e5b2ef4) * kernel (= 4.9.37-1-7f0de30d5b73958cb146494d8e5b2ef4) *
* opkg_install_cmd: Cannot install package ppp-mod-pptp.
* satisfy_dependencies_for: Cannot satisfy the following dependencies for kmod-nf-nathelper-extra:
* kernel (= 4.9.37-1-7f0de30d5b73958cb146494d8e5b2ef4) * kernel (= 4.9.37-1-7f0de30d5b73958cb146494d8e5b2ef4) *
* opkg_install_cmd: Cannot install package kmod-nf-nathelper-extra.
root@LEDE:~# #is for routing vpn client to vpn client and lan to vpn-client
root@LEDE:~# opkg install kmod-ipt-nat-extra
Package kmod-ipt-nat-extra (4.9.37-1) installed in root is up to date.
root@LEDE:~#

and:

root@LEDE:~# #traffic shaping:
root@LEDE:~# #############################
root@LEDE:~# opkg install luci-app-sqm sqm-scripts kmod-sched-cake #–force-reinstall
Installing luci-app-sqm (1.1.3-1) to root...
Downloading http://downloads.lede-project.org/snapshots/packages/mipsel_24kc/packages/luci-app-sqm_1.1.3-1_all.ipk Installing sqm-scripts (1.1.3-1) to root...
Downloading http://downloads.lede-project.org/snapshots/packages/mipsel_24kc/packages/sqm-scripts_1.1.3-1_all.ipk Package kmod-sched-cake (4.9.37+2017-01-28-9789742c-1) installed in root is up to date.
Configuring sqm-scripts.
Collected errors:
* satisfy_dependencies_for: Cannot satisfy the following dependencies for luci-app-sqm:
* kernel (= 4.9.37-1-7f0de30d5b73958cb146494d8e5b2ef4) *
* opkg_install_cmd: Cannot install package luci-app-sqm.
root@LEDE:~#
root@LEDE:~# #loadbalancing:

and

Collected errors:
* satisfy_dependencies_for: Cannot satisfy the following dependencies for conntrackd:
* kernel (= 4.9.37-1-7f0de30d5b73958cb146494d8e5b2ef4) *
* opkg_install_cmd: Cannot install package conntrackd.

strange , isn'T it ?
or is lede currently building some new packages ?

all other packages are correct now installed (and i have a lot as you know ....)
:)

camel commented on 24.07.2017 17:43

Pls make new image with new
kernel (= 4.9.37-1-7f0de30d5b73958cb146494d8e5b2ef4
Lede was building new images/packages

Pierre Parent commented on 24.07.2017 18:05

No the kernel did not change, it's just that I suppressed Zhong Jianxin's technique to cheat on kernel magic numbers. Because otherwise you can install modules that are not correctly compiled for our kernel, and are very unstable, as you experienced.

I will make a new image with luci-app-sqm sqm-scripts , so that you test with traffic shaping as it has a very important impact on rcu_sched problems.

Note also that you can easily cross-compile any other package you need with sdk I posted above, basically in 3 commands;

./scripts/feeds update -a 
./scripts/feeds install <packagename>
make package/<packagename>/compile

https://wiki.openwrt.org/doc/howto/obtain.firmware.sdk

camel commented on 24.07.2017 18:49

of course, a new build was made..
That is the reason, that now I can't install free kernel modules.
Before it was build from Sunday, now a new one from Monday(today)

Zhong Jianxin commented on 25.07.2017 02:18
  @Zhong Jianxin: will it work even with a modified kernel (since we change patches used in upstream)?
It depends on the modification, in this case, it should work.

The reason why there are so many `unknown symbol` is that some kernel configs are only enabled when certain packages selected, e.g. selecting kmod-fs-f2fs will enable this kernel configs:

  CONFIG_F2FS_FS
  CONFIG_F2FS_STAT_FS=y
  CONFIG_F2FS_FS_XATTR=y
  CONFIG_F2FS_FS_POSIX_ACL=n
  CONFIG_F2FS_FS_SECURITY=n
  CONFIG_F2FS_CHECK_FS=n

Upstream snapshot build will select many packages, but it's probably not the case of custom build.

Here is another way to build as close as upstream snapshot:

  $ wget https://downloads.lede-project.org/snapshots/targets/ramips/mt7621/config.seed -O .config
  $ make defconfig
  $ make dirclean
  $ make

But it will take much longer time.

Just tested this, it built the some kernel version as current upstream snapshot, without overriding LINUX_VERMAGIC.

camel commented on 25.07.2017 06:10

@piere: possible to prepare a new build with current new kernel magic id ?

Pierre Parent commented on 25.07.2017 06:41

I will not use the magic number technique again because:

1- It's useless, since modules that were not included in my build, and therefore not built will not work @Zhong Jianxin said. Those who are built I can include them directly in the image.

2- It can lead to wrong diagnostic and wrong conclusions for our testing. Since as we saw we can get kernel errors due to incompatibility between the kernel I built and modules in packages upstream.

3- I will compile an image with traffic shaping in one hour, so that you can test with traffic shaping.

camel commented on 25.07.2017 06:57

Hmm, not sure, if you can add the pptpd + pppd packages included, too.
As that would be mandatory for me.
Anyhow, I will try if I can use it for testing anyhow.

Pierre Parent commented on 25.07.2017 08:38

@Camel can you please test with QOS/SQM enabled on the version I just sent to my server?

As that would be mandatory for me.
>Anyhow, I will try if I can use it for testing anyhow

Anyways you should not use my images for anything else than testing....

camel commented on 25.07.2017 09:07

sure, i will try ...
i will try to install now

camel commented on 25.07.2017 09:11

pls give me the DL link,
because the other link shows me the identically version like yesterday ...
i DL: https://www.own-mailbox.com/lede/lede-ramips-mt7621-zbt-wg3526-16M-squashfs-sysupgrade.bin is the same like yesterday ...
pls DL link ?

Pierre Parent commented on 25.07.2017 09:22

Hum no it should not be, it corresponds to the image I compiled today, with luci-app-sqm sqm-scripts

Md5sum: 71121b4a6a30abd6627595d01bd0374c

camel commented on 25.07.2017 09:29

sorry, my mistake ...
but you forgot to set kernel magic ....

as i can'T install pptp client + server packages ...

root@LEDE:~# #pptp + pptpd ⇒ root@LEDE:~# #

root@LEDE:~# opkg install kmod-crypto-manager kmod-crypto-pcompress kmod-crypto-ecb

opkg install ppp-mod-pptp kmod-nf-nathelper-extra
#is for routing vpn client to vpn client and lan to vpn-client
opkg install kmod-ipt-nat-extra

#notfalls via:
# opkg install ppp-mod-pptp kmod-nf-nathelper-extra #–force-depends –force-reinstall
#If LuCI support is desired, additionally install the protocol package:
opkg install luci-proto-ppp

#VPN PPTP server:
opkg install pptpd kmod-mppe
Package kmod-crypto-manager (4.9.37-1) installed in root is up to date.
Package kmod-crypto-pcompress (4.9.37-1) installed in root is up to date.
Package kmod-crypto-ecb (4.9.37-1) installed in root is up to date.
root@LEDE:~# opkg install kmod-crypto-null kmod-crypto-aead kmod-iptunnel kmod-gre resolveip kmod-pptp kmod-crypto-sha1
Package kmod-crypto-null (4.9.37-1) installed in root is up to date.
Package kmod-crypto-aead (4.9.37-1) installed in root is up to date.
Installing kmod-iptunnel (4.9.37-1) to root...
Downloading http://downloads.lede-project.org/snapshots/targets/ramips/mt7621/packages/kmod-iptunnel_4.9.37-1_mipsel_24kc.ipk Installing kmod-gre (4.9.37-1) to root...
Downloading http://downloads.lede-project.org/snapshots/targets/ramips/mt7621/packages/kmod-gre_4.9.37-1_mipsel_24kc.ipk Installing resolveip (2) to root...
Downloading http://downloads.lede-project.org/snapshots/packages/mipsel_24kc/base/resolveip_2_mipsel_24kc.ipk Installing kmod-pptp (4.9.37-1) to root...
Downloading http://downloads.lede-project.org/snapshots/targets/ramips/mt7621/packages/kmod-pptp_4.9.37-1_mipsel_24kc.ipk Installing kmod-crypto-sha1 (4.9.37-1) to root...
Downloading http://downloads.lede-project.org/snapshots/targets/ramips/mt7621/packages/kmod-crypto-sha1_4.9.37-1_mipsel_24kc.ipk Configuring resolveip.
Collected errors:
* satisfy_dependencies_for: Cannot satisfy the following dependencies for kmod-iptunnel:
* kernel (= 4.9.37-1-7f0de30d5b73958cb146494d8e5b2ef4) *
* opkg_install_cmd: Cannot install package kmod-iptunnel.
* satisfy_dependencies_for: Cannot satisfy the following dependencies for kmod-gre:
* kernel (= 4.9.37-1-7f0de30d5b73958cb146494d8e5b2ef4) *
* opkg_install_cmd: Cannot install package kmod-gre.
* satisfy_dependencies_for: Cannot satisfy the following dependencies for kmod-pptp:
* kernel (= 4.9.37-1-7f0de30d5b73958cb146494d8e5b2ef4) *
* opkg_install_cmd: Cannot install package kmod-pptp.
* satisfy_dependencies_for: Cannot satisfy the following dependencies for kmod-crypto-sha1:
* kernel (= 4.9.37-1-7f0de30d5b73958cb146494d8e5b2ef4) *
* opkg_install_cmd: Cannot install package kmod-crypto-sha1.
root@LEDE:~#
root@LEDE:~# opkg install ppp-mod-pptp kmod-nf-nathelper-extra
Installing ppp-mod-pptp (2.4.7-12) to root...
Downloading http://downloads.lede-project.org/snapshots/packages/mipsel_24kc/base/ppp-mod-pptp_2.4.7-12_mipsel_24kc.ipk Installing kmod-nf-nathelper-extra (4.9.37-1) to root...
Downloading http://downloads.lede-project.org/snapshots/targets/ramips/mt7621/packages/kmod-nf-nathelper-extra_4.9.37-1_mipsel_24kc.ipk Collected errors:
* satisfy_dependencies_for: Cannot satisfy the following dependencies for ppp-mod-pptp:
* kernel (= 4.9.37-1-7f0de30d5b73958cb146494d8e5b2ef4) * kernel (= 4.9.37-1-7f0de30d5b73958cb146494d8e5b2ef4) * kernel (= 4.9.37-1-7f0de30d5b73958cb146494d8e5b2ef4) * kernel (= 4.9.37-1-7f0de30d5b73958cb146494d8e5b2ef4) * kernel (= 4.9.37-1-7f0de30d5b73958cb146494d8e5b2ef4) *
* opkg_install_cmd: Cannot install package ppp-mod-pptp.
* satisfy_dependencies_for: Cannot satisfy the following dependencies for kmod-nf-nathelper-extra:
* kernel (= 4.9.37-1-7f0de30d5b73958cb146494d8e5b2ef4) * kernel (= 4.9.37-1-7f0de30d5b73958cb146494d8e5b2ef4) *
* opkg_install_cmd: Cannot install package kmod-nf-nathelper-extra.
root@LEDE:~# #is for routing vpn client to vpn client and lan to vpn-client
root@LEDE:~# opkg install kmod-ipt-nat-extra
Package kmod-ipt-nat-extra (4.9.37-1) installed in root is up to date.
root@LEDE:~#
root@LEDE:~# #notfalls via:
root@LEDE:~# # opkg install ppp-mod-pptp kmod-nf-nathelper-extra #–force-depends –force-reinstall
root@LEDE:~# #If LuCI support is desired, additionally install the protocol package:
root@LEDE:~# opkg install luci-proto-ppp
Upgrading luci-proto-ppp on root from git-17.205.18138-17de308-1 to git-17.205.60871-6c8a9f1-1...
Downloading http://downloads.lede-project.org/snapshots/packages/mipsel_24kc/luci/luci-proto-ppp_git-17.205.60871-6c8a9f1-1_all.ipk Configuring luci-proto-ppp.
root@LEDE:~#
root@LEDE:~# #VPN PPTP server:
root@LEDE:~# opkg install pptpd kmod-mppe
Installing pptpd (1.4.0-3) to root...
Downloading http://downloads.lede-project.org/snapshots/packages/mipsel_24kc/packages/pptpd_1.4.0-3_mipsel_24kc.ipk Installing kmod-mppe (4.9.37-1) to root...
Downloading http://downloads.lede-project.org/snapshots/targets/ramips/mt7621/packages/kmod-mppe_4.9.37-1_mipsel_24kc.ipk Configuring kmod-mppe.
1 module could not be probed
- ppp_mppe
Collected errors:
* satisfy_dependencies_for: Cannot satisfy the following dependencies for pptpd:
* kernel (= 4.9.37-1-7f0de30d5b73958cb146494d8e5b2ef4) * kernel (= 4.9.37-1-7f0de30d5b73958cb146494d8e5b2ef4) * kernel (= 4.9.37-1-7f0de30d5b73958cb146494d8e5b2ef4) * kernel (= 4.9.37-1-7f0de30d5b73958cb146494d8e5b2ef4) *
* opkg_install_cmd: Cannot install package pptpd.
root@LEDE:~#

Pierre Parent commented on 25.07.2017 10:00
but you forgot to set kernel magic ....

I did not forget I purposly did not do it for reasons mentioned above.

camel commented on 25.07.2017 10:35

hmm, but i really need the pptp stuff .. otherwise i can't really test it more in detail
as i have there to connect via pptp
can you add it ?
or can you make 1 version which is having the kernel magic option on compile ?
+ if possible - pls without the SSL package for luci, as i need to use a different one .. otherwise i can't install the openvpn ssl package.

or if kernel magic set to lede trunk snapshot, then i can remove and install the real one which i need...

as
#WEBIFACE:
opkg install luci #–force-reinstall
#opkg install luci-ssl #–force-reinstall
opkg install luci-ssl-openssl

#OPENVPN: (ca. 1MB space needed)
opkg install openvpn-openssl luci-app-openvpn
#opkg install openvpn-mbedtls luci-app-openvpn
opkg install nano libustream-openssl ca-bundle ca-certificates

camel commented on 25.07.2017 10:54

meanwhile i tested ... hmm ...seems to be, that I'M getting a mem issue ...
not sure, what is all included ... but i guess it is not related to the RCU_SCHED errors ...

maybe that was related to a "speedtest.py" which i run every 5min to have the statistics about the router internet speed ...
also i have seen you are having squid installed and so on ...

Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.520000] luci invoked oom-killer: gfp_mask=0x27000c0(GFP_KERNEL_ACCOUNT|GFP_NOTRACK), nodemask=0, order=1, oom_score_adj=0
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.530000] COMPACTION is disabled!!!
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.540000] CPU: 2 PID: 32685 Comm: luci Not tainted 4.9.37 #0
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.540000] Stack : 00000000 00000000 80537b2a 00000032 803f4084 00000000 00000000 80530000
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.550000] 81fa462c 804d7da7 8046dff0 00000002 00007fad 80533824 00000001 00200000
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.560000] 00001321 80069890 00000000 800696b0 00000000 00000004 80472c00 82745c3c
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.570000] 00000000 800a5d98 00000000 00000000 80537b2a 00000000 82745d28 00745c3c
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.580000] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.580000] ...
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.590000] Call Trace:
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.590000] [<8000f644>] show_stack+0x54/0x88
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.590000] [<801e5924>] dump_stack+0x84/0xc0
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.600000] [<800ea424>] dump_header.isra.4+0x84/0x1b4
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.600000] [<800ac264>] oom_kill_process+0xd0/0x484
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.610000] [<800acb40>] out_of_memory+0x3bc/0x3fc
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.610000] [<800b01c4>]
alloc_pages_nodemask+0x5e4/0xa58
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.620000] [<800281b0>] copy_process.isra.8.part.9+0x10c/0x1300
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.620000] [<80029520>] _do_fork+0xcc/0x2d8
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.630000] [<800297dc>] SyS_clone+0x20/0x2c
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.630000] [<80016558>] syscall_common+0x34/0x58
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.640000] Mem-Info:
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.640000] active_anon:13977 inactive_anon:1643 isolated_anon:0
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.640000] active_file:805 inactive_file:2995 isolated_file:0
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.640000] unevictable:1 dirty:2 writeback:0 unstable:0
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.640000] slab_reclaimable:3926 slab_unreclaimable:35491
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.640000] mapped:3433 shmem:8024 pagetables:227 bounce:0
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.640000] free:49832 free_pcp:28 free_cma:0
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.670000] Node 0 active_anon:55908kB inactive_anon:6572kB active_file:3220kB inactive_file:11980kB unevictable:4kB isolated(anon):0kB isolated(file):0kB mapped:13732kB dirty:8kB writeback:0kB shmem:32096kB writeback_tmp:0kB unstable:0kB pages_scanned:0 all_unreclaimable? no
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.700000] Normal free:20008kB min:16384kB low:20480kB high:24576kB active_anon:0kB inactive_anon:0kB active_file:4kB inactive_file:84kB unevictable:0kB writepending:8kB present:262144kB managed:251688kB mlocked:0kB slab_reclaimable:15704kB slab_unreclaimable:141964kB kernel_stack:56192kB pagetables:908kB bounce:0kB free_pcp:232kB local_pcp:0kB free_cma:0kB
Tue Jul 25 12:50:05 2017 kern.emerg kernel: lowmem_reserve[]: 0 2048 2048
Tue Jul 25 12:50:05 2017 kern.warn kernel: [ 1065.730000] HighMem free:179216kB min:256kB low:4520kB high:8784kB active_anon:55908kB inactive_anon:6572kB active_file:3148kB inactive_file:11888kB unevictable:4kB writepending:0kB present:262144kB managed:262144kB mlocked:4kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:52kB local_pcp:0kB free_cma:0kB
Tue Jul 25 12:50:05 2017 kern.emerg kernel: lowmem_reserve[]: 0 0 0
Tue Jul 25 12:50:05 2017 kern.warn kernel: [ 1065.770000] Normal: 23*4kB (UMH) 34*8kB (UMEH) 19*16kB (UMH) 25*32kB (UMEH) 16*64kB (UME) 9*128kB (UMEH) 4*256kB (UME) 6*512kB (UM) 6*1024kB (UMEH) 3*2048kB (UME) 0*4096kB = 20028kB
Tue Jul 25 12:50:05 2017 kern.emerg kernel: HighMem: 881*4kB (UM) 957*8kB (UM) 543*16kB (UM) 360*32kB (UM) 133*64kB (UM) 18*128kB (M) 8*256kB (UM) 3*512kB (M) 4*1024kB (M) 1*2048kB (U) 31*4096kB (M) = 178908kB
Tue Jul 25 12:50:05 2017 kern.emerg kernel: 11849 total pagecache pages
Tue Jul 25 12:50:05 2017 kern.warn kernel: [ 1065.800000] 0 pages in swap cache
Tue Jul 25 12:50:05 2017 kern.warn kernel: [ 1065.810000] Swap cache stats: add 0, delete 0, find 0/0
Tue Jul 25 12:50:05 2017 kern.warn kernel: [ 1065.810000] Free swap = 0kB
Tue Jul 25 12:50:05 2017 kern.warn kernel: [ 1065.810000] Total swap = 0kB
Tue Jul 25 12:50:05 2017 kern.warn kernel: [ 1065.820000] 131072 pages RAM
Tue Jul 25 12:50:05 2017 kern.warn kernel: [ 1065.820000] 65536 pages HighMem/MovableOnly
Tue Jul 25 12:50:05 2017 kern.warn kernel: [ 1065.820000] 2614 pages reserved
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1065.830000] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1065.840000] [ 731] 0 731 298 149 3 0 0 0 ubusd
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1065.840000] [ 738] 0 738 225 123 3 0 0 0 askfirst
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1065.850000] [ 1317] 0 1317 823 694 5 0 0 0 logd
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1065.860000] [ 1318] 0 1318 336 198 4 0 0 0 logread
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1065.870000] [ 1330] 0 1330 381 213 4 0 0 0 rpcd
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1065.880000] [ 1499] 0 1499 435 257 4 0 0 0 netifd
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1065.890000] [ 1548] 0 1548 356 183 4 0 0 0 odhcpd
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1065.900000] [ 1581] 0 1581 300 202 3 0 0 0 crond
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1065.910000] [ 1603] 0 1603 267 153 4 0 0 0 dropbear
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1065.910000] [ 1695] 0 1695 299 153 3 0 0 0 udhcpc
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1065.920000] [ 1703] 0 1703 256 139 3 0 0 0 odhcp6c
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1065.930000] [ 1762] 65536 1762 963 497 4 0 0 0 lighttpd
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1065.940000] [ 1964] 0 1964 633 413 4 0 0 0 uhttpd
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1065.950000] [ 1996] 0 1996 242 10 3 0 0 0 vsftpd
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1065.960000] [ 2029] 0 2029 751 180 4 0 0 0 keepalived
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1065.970000] [ 2045] 0 2045 759 37 5 0 0 0 keepalived
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1065.980000] [ 2237] 0 2237 280 139 3 0 0 0 pppd
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1065.990000] [ 2286] 453 2286 269 203 3 0 0 0 dnsmasq
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1065.990000] [ 2323] 0 2323 420 26 4 0 0 0 hostapd
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.000000] [ 2325] 0 2325 420 26 3 0 0 0 hostapd
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.010000] [ 2447] 0 2447 1462 564 5 0 0 0 collectd
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.020000] [ 2488] 0 2488 228 9 3 0 0 0 ntpclient
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.030000] [ 2535] 65534 2535 300 220 4 0 0 0 collectd_ifacei
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.040000] [ 2541] 65534 2541 301 236 3 0 0 0 collectd_gsminf
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.050000] [ 2542] 65534 2542 300 235 3 0 0 0 collectd_mwan-s
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.060000] [ 2543] 65534 2543 300 236 4 0 0 0 collectd_speedt
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.070000] [ 2635] 0 2635 3576 2803 8 0 0 0 squid
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.080000] [ 3059] 0 3059 317 253 3 0 0 0 mwan3track
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.090000] [ 3945] 0 3945 333 227 4 0 0 0 luci-bwc
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.090000] [ 4547] 0 4547 284 9 5 0 0 0 dropbear
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.100000] [ 4683] 0 4683 300 235 4 0 0 0 ash
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.110000] [10425] 0 10425 306 216 3 0 0 0 smsd
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.120000] [10429] 0 10429 315 64 4 0 0 0 smsd
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.130000] [10631] 0 10631 244 167 3 0 0 0 vnstatd
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.140000] [12948] 0 12948 284 9 5 0 0 0 dropbear
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.150000] [12962] 0 12962 300 236 4 0 0 0 ash
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.150000] [18822] 0 18822 842 733 5 0 0 0 openvpn
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.160000] [19980] 0 19980 284 9 5 0 0 0 dropbear
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.170000] [20365] 0 20365 300 154 5 0 0 0 ash
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.180000] [11049] 65534 11049 300 10 4 0 0 0 collectd_speedt
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.190000] [11050] 0 11050 5266 4606 9 0 0 0 python
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.200000] [12184] 0 12184 300 139 4 0 0 0 sleep
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.210000] [12380] 65534 12380 301 10 3 0 0 0 collectd_gsminf
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.220000] [12381] 0 12381 303 154 4 0 0 0 3gsignal
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.230000] [12382] 65534 12382 299 138 3 0 0 0 sed
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.240000] [12430] 0 12430 243 145 4 0 0 0 gcom
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.240000] [16962] 65534 16962 299 137 3 0 0 0 sleep
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.250000] [19601] 65534 19601 299 154 3 0 0 0 sleep
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.260000] [24961] 0 24961 336 199 3 0 0 0 logread
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.270000] [25709] 0 25709 310 154 4 0 0 0 hotplug-call
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.280000] [25717] 0 25717 337 229 4 0 0 0 hotplug-call
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.290000] [25747] 0 25747 278 144 4 0 0 0 ipset
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.300000] [26138] 0 26138 842 728 4 0 0 0 openvpn
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.310000] [32683] 0 32683 667 419 3 0 0 0 luci
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.310000] [32684] 0 32684 667 416 4 0 0 0 luci
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.320000] [32685] 0 32685 667 418 6 0 0 0 luci
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.330000] [32687] 0 32687 257 176 4 0 0 0 modprobe
Tue Jul 25 12:50:05 2017 kern.info kernel: [ 1066.340000] [32740] 0 32740 299 138 3 0 0 0 sh
Tue Jul 25 12:50:05 2017 kern.err kernel: [ 1066.350000] Out of memory: Kill process 11050 (python) score 34 or sacrifice child
Tue Jul 25 12:50:05 2017 kern.err kernel: [ 1066.360000] Killed process 11050 (python) total-vm:21064kB, anon-rss:14176kB, file-rss:4244kB, shmem-rss:4kB

for traffic shapping i would need the pptp in any case, as the VPN's are shapped ..

camel commented on 25.07.2017 13:32

meanwhile .. i tested with traffic shapping ... and produced a lot of traff ic on:
.) LAN _⇒ ~10GB transfer
.) WLAN 5Ghz _⇒ ~15GB transfer
.) WLAN 2,4Ghz ⇒ ~5GB transfer

and got few errors ....

Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.520000] luci invoked oom-killer: gfp_mask=0x27000c0(GFP_KERNEL_ACCOUNT|__GFP_NOTRACK), nodemask=0, order=1, oom_score_adj=0
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.530000] COMPACTION is disabled!!!
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.540000] CPU: 2 PID: 32685 Comm: luci Not tainted 4.9.37 #0
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.540000] Stack : 00000000 00000000 80537b2a 00000032 803f4084 00000000 00000000 80530000
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.550000]         81fa462c 804d7da7 8046dff0 00000002 00007fad 80533824 00000001 00200000
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.560000]         00001321 80069890 00000000 800696b0 00000000 00000004 80472c00 82745c3c
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.570000]         00000000 800a5d98 00000000 00000000 80537b2a 00000000 82745d28 00745c3c
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.580000]         00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.580000]         ...
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.590000] Call Trace:
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.590000] [<8000f644>] show_stack+0x54/0x88
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.590000] [<801e5924>] dump_stack+0x84/0xc0
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.600000] [<800ea424>] dump_header.isra.4+0x84/0x1b4
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.600000] [<800ac264>] oom_kill_process+0xd0/0x484
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.610000] [<800acb40>] out_of_memory+0x3bc/0x3fc
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.610000] [<800b01c4>] __alloc_pages_nodemask+0x5e4/0xa58
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.620000] [<800281b0>] copy_process.isra.8.part.9+0x10c/0x1300
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.620000] [<80029520>] _do_fork+0xcc/0x2d8
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.630000] [<800297dc>] SyS_clone+0x20/0x2c
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.630000] [<80016558>] syscall_common+0x34/0x58
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.640000] Mem-Info:
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.640000] active_anon:13977 inactive_anon:1643 isolated_anon:0
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.640000]  active_file:805 inactive_file:2995 isolated_file:0
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.640000]  unevictable:1 dirty:2 writeback:0 unstable:0
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.640000]  slab_reclaimable:3926 slab_unreclaimable:35491
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.640000]  mapped:3433 shmem:8024 pagetables:227 bounce:0
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.640000]  free:49832 free_pcp:28 free_cma:0
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.670000] Node 0 active_anon:55908kB inactive_anon:6572kB active_file:3220kB inactive_file:11980kB unevictable:4kB isolated(anon):0kB isolated(file):0kB mapped:13732kB dirty:8kB writeback:0kB shmem:32096kB writeback_tmp:0kB unstable:0kB pages_scanned:0 all_unreclaimable? no
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.700000] Normal free:20008kB min:16384kB low:20480kB high:24576kB active_anon:0kB inactive_anon:0kB active_file:4kB inactive_file:84kB unevictable:0kB writepending:8kB present:262144kB managed:251688kB mlocked:0kB slab_reclaimable:15704kB slab_unreclaimable:141964kB kernel_stack:56192kB pagetables:908kB bounce:0kB free_pcp:232kB local_pcp:0kB free_cma:0kB
Tue Jul 25 12:50:05 2017 kern.emerg kernel: lowmem_reserve[]: 0 2048 2048


Tue Jul 25 13:13:39 2017 kern.err kernel: [ 2479.840000] INFO: rcu_sched detected stalls on CPUs/tasks:
Tue Jul 25 13:13:39 2017 kern.err kernel: [ 2479.840000] 	3-...: (0 ticks this GP) idle=c95/140000000000000/0 softirq=302510/302510 fqs=1 
Tue Jul 25 13:13:39 2017 kern.err kernel: [ 2479.850000] 	(detected by 2, t=6003 jiffies, g=66650, c=66649, q=150961)
Tue Jul 25 13:13:39 2017 kern.info kernel: [ 2479.860000] Task dump for CPU 3:
Tue Jul 25 13:13:39 2017 kern.info kernel: [ 2479.860000] luci            R  running task        0 21284   1964 0x08100000
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.870000] Stack : 00000000 8baf8c38 80528b80 8c0a9a98 8c0a9a98 8fc02380 8d0fe180 000000bb
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.870000]         8fc02680 8ecc0080 8ecc2180 800e5fd0 8ecc2180 800e79c0 00000001 811dc800
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.880000]         8c0a9b20 800af53c 00000000 804e0000 8149499c ffffffff 00000001 00008159
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.890000]         8d5676a8 0000005a 00000058 800e55c0 024102c0 802d9e80 00210d00 804d55f4
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.900000]         00000000 800afce0 026012d0 80479f14 811dc800 0000000f 00210d00 804d55f4
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.910000]         ...
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.910000] Call Trace:
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.910000] [<803dc338>] __schedule+0x5d4/0x7a4
Tue Jul 25 13:13:39 2017 kern.err kernel: [ 2479.920000] rcu_sched kthread starved for 2050 jiffies! g66650 c66649 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1
Tue Jul 25 13:13:39 2017 kern.info kernel: [ 2479.930000] rcu_sched       S    0     7      2 0x00100000
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.930000] Stack : 814a9bc0 8fc61250 8fc89e50 814b4320 8fc89e50 00000000 00000001 8007b658
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.940000]         814b4320 00034b87 814b4320 8fc89e50 804d0000 00000000 804e0000 804d0000
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.950000]         804dd5f0 803dc574 00034b87 804d0000 8fc89e50 814b4320 00034b87 803deed8
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.960000]         804d55f4 804e0000 00000001 804d0000 814b6240 00000001 00000000 814b4710
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.970000]         00034b87 8007bb54 8fc61250 39800003 804dd4c0 00000001 804d0000 00000000
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.970000]         ...
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.980000] Call Trace:
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.980000] [<803dc338>] __schedule+0x5d4/0x7a4
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.980000] [<803dc574>] schedule+0x6c/0x84
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.990000] [<803deed8>] schedule_timeout+0x160/0x19c
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.990000] [<80079090>] rcu_gp_kthread+0x7f4/0x7fc
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2480.000000] [<80044be8>] kthread+0xd8/0xec
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2480.000000] [<8000a318>] ret_from_kernel_thread+0x14/0x1c

Tue Jul 25 13:17:39 2017 kern.err kernel: [ 2720.280000] INFO: rcu_sched detected stalls on CPUs/tasks:
Tue Jul 25 13:17:39 2017 kern.err kernel: [ 2720.280000] 	3-...: (0 ticks this GP) idle=9c0/0/0 softirq=302510/302510 fqs=1 
Tue Jul 25 13:17:39 2017 kern.err kernel: [ 2720.290000] 	(detected by 1, t=6003 jiffies, g=66656, c=66655, q=340412)
Tue Jul 25 13:17:39 2017 kern.info kernel: [ 2720.290000] Task dump for CPU 3:
Tue Jul 25 13:17:39 2017 kern.info kernel: [ 2720.300000] swapper/3       R  running task        0     0      1 0x00100000
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.300000] Stack : 00000068 8fc62498 00000001 84ca6177 00000279 00000000 0000022e 009a3220
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.310000]         804d548c 00000008 80472554 804e0000 00000001 00000001 bfbf8000 00000003
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.320000]         00000000 803dc574 00000000 00000000 8fc94000 804d548c 00000008 800c5534
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.330000]         804d548c 00000008 80472554 804e0000 804d548c 8006304c 00000001 00000001
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.340000]         00000001 00000001 00000001 00000001 00000001 00000001 00000001 9fc0046c
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.350000]         ...
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.350000] Call Trace:
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.350000] [<803dc338>] __schedule+0x5d4/0x7a4
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.360000] [<803dc574>] schedule+0x6c/0x84
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.360000] [<800c5534>] quiet_vmstat+0x54/0x80
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.370000] [<8006304c>] cpu_startup_entry+0xec/0x150
Tue Jul 25 13:17:39 2017 kern.err kernel: [ 2720.370000] rcu_sched kthread starved for 2496 jiffies! g66656 c66655 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1
Tue Jul 25 13:17:39 2017 kern.info kernel: [ 2720.380000] rcu_sched       S    0     7      2 0x00100000
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.390000] Stack : 814a9bc0 8fc61250 8fc89e50 814b4320 8fc89e50 804d0000 00000001 8007b658
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.390000]         814b4320 0003a7b6 814b4320 8fc89e50 804d0000 00000000 804e0000 804d0000
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.400000]         804dd5f0 803dc574 0003a7b6 804d0000 8fc89e50 814b4320 0003a7b6 803deed8
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.410000]         804d55f4 804e0000 00000001 804d0000 814b6240 00000001 00000000 814b46c8
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.420000]         0003a7b6 8007bb54 8fc61250 35000003 804dd4c0 00000001 804d0000 00000000
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.430000]         ...
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.430000] Call Trace:
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.430000] [<803dc338>] __schedule+0x5d4/0x7a4
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.440000] [<803dc574>] schedule+0x6c/0x84
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.440000] [<803deed8>] schedule_timeout+0x160/0x19c
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.450000] [<80079090>] rcu_gp_kthread+0x7f4/0x7fc
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.450000] [<80044be8>] kthread+0xd8/0xec
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.460000] [<8000a318>] ret_from_kernel_thread+0x14/0x1c


Tue Jul 25 13:19:54 2017 kern.err kernel: [ 2855.490000] INFO: rcu_sched detected stalls on CPUs/tasks:
Tue Jul 25 13:19:54 2017 kern.err kernel: [ 2855.490000] 	3-...: (0 ticks this GP) idle=69c/0/0 softirq=302510/302510 fqs=1 
Tue Jul 25 13:19:54 2017 kern.err kernel: [ 2855.500000] 	(detected by 1, t=6003 jiffies, g=66673, c=66672, q=275515)
Tue Jul 25 13:19:54 2017 kern.info kernel: [ 2855.500000] Task dump for CPU 3:
Tue Jul 25 13:19:54 2017 kern.info kernel: [ 2855.510000] swapper/3       R  running task        0     0      1 0x00100000
Tue Jul 25 13:19:54 2017 kern.warn kernel: [ 2855.510000] Stack : 00000068 00000001 00000001 001fd953 00000000 77edc2c0 00000000 00000000
Tue Jul 25 13:19:54 2017 kern.warn kernel: [ 2855.520000]         804d548c 00000008 80472554 804e0000 00000001 00000001 bfbf8000 00000003
Tue Jul 25 13:19:54 2017 kern.warn kernel: [ 2855.530000]         00000000 8000c1cc 00000000 00000000 8fc94000 8fc95f00 00000000 80063054
Tue Jul 25 13:19:54 2017 kern.warn kernel: [ 2855.540000]         1100fc03 00000001 00000001 804d548c 804d548c 8006304c 00000001 00000001
Tue Jul 25 13:19:54 2017 kern.warn kernel: [ 2855.550000]         00000001 00000001 00000001 00000001 00000001 00000001 00000001 9fc0046c
Tue Jul 25 13:19:54 2017 kern.warn kernel: [ 2855.560000]         ...
Tue Jul 25 13:19:54 2017 kern.warn kernel: [ 2855.560000] Call Trace:
Tue Jul 25 13:19:55 2017 kern.warn kernel: [ 2855.560000] [<803dc338>] __schedule+0x5d4/0x7a4
Tue Jul 25 13:19:55 2017 kern.warn kernel: [ 2855.570000] [<8000c1cc>] r4k_wait_irqoff+0x0/0x20
Tue Jul 25 13:19:55 2017 kern.err kernel: [ 2855.570000] rcu_sched kthread starved for 3051 jiffies! g66673 c66672 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1
Tue Jul 25 13:19:55 2017 kern.info kernel: [ 2855.580000] rcu_sched       S    0     7      2 0x00100000
Tue Jul 25 13:19:55 2017 kern.warn kernel: [ 2855.590000] Stack : 814b5bc0 8fc61250 8fc89e50 814b4320 8fc89e50 804d0000 00000001 8007b658
Tue Jul 25 13:19:55 2017 kern.warn kernel: [ 2855.590000]         814b4320 0003da5b 814b4320 8fc89e50 804d0000 00000000 804e0000 804d0000
Tue Jul 25 13:19:55 2017 kern.warn kernel: [ 2855.600000]         804dd5f0 803dc574 0003da5b 804d0000 8fc89e50 814b4320 0003da5b 803deed8
Tue Jul 25 13:19:55 2017 kern.warn kernel: [ 2855.610000]         804d55f4 804e0000 00000001 804d0000 814b6240 00000001 00000000 814b4870
Tue Jul 25 13:19:55 2017 kern.warn kernel: [ 2855.620000]         0003da5b 8007bb54 8fc61250 4f800003 804dd4c0 00000001 804d0000 00000000
Tue Jul 25 13:19:55 2017 kern.warn kernel: [ 2855.630000]         ...
Tue Jul 25 13:19:55 2017 kern.warn kernel: [ 2855.630000] Call Trace:
Tue Jul 25 13:19:55 2017 kern.warn kernel: [ 2855.630000] [<803dc338>] __schedule+0x5d4/0x7a4
Tue Jul 25 13:19:55 2017 kern.warn kernel: [ 2855.640000] [<803dc574>] schedule+0x6c/0x84
Tue Jul 25 13:19:55 2017 kern.warn kernel: [ 2855.640000] [<803deed8>] schedule_timeout+0x160/0x19c
Tue Jul 25 13:19:55 2017 kern.warn kernel: [ 2855.650000] [<80079090>] rcu_gp_kthread+0x7f4/0x7fc
Tue Jul 25 13:19:55 2017 kern.warn kernel: [ 2855.650000] [<80044be8>] kthread+0xd8/0xec
Tue Jul 25 13:19:55 2017 kern.warn kernel: [ 2855.660000] [<8000a318>] ret_from_kernel_thread+0x14/0x1c

Tue Jul 25 13:20:21 2017 daemon.notice squid[2635]: Logfile: opening log stdio:/var/logs/netdb.state
Tue Jul 25 13:20:21 2017 daemon.notice squid[2635]: ERROR: logfileOpen stdio:/var/logs/netdb.state: (2) No such file or directory
Tue Jul 25 13:20:21 2017 daemon.notice squid[2635]: logfileOpen: stdio:/var/logs/netdb.state: couldn't open!
Tue Jul 25 13:20:21 2017 daemon.notice squid[2635]: netdbSaveState: stdio:/var/logs/netdb.state: (2) No such file or directory
Pierre Parent commented on 25.07.2017 17:11

We need a better patch...
Hope John will be able to help us...

Admin
john commented on 25.07.2017 17:20

i've done nothing all day but play around with this. i am unfortunately not able to reproduce this issue. i've just sent my latest version of the patch to someone for testing. lets hope for the best.

Pierre Parent commented on 25.07.2017 17:32

@john thanks a lot.

Maybe if you send us your latest version of your patch we can test it too.

dchard commented on 25.07.2017 18:16

@john: I agree with Pierre: if you can send us a build with your (latest) patches inside, we are happy to test. Like we did with previous versions :-)

Pierre Parent commented on 25.07.2017 18:20

If you don't have a build I can do the build as I did for last hint.

Admin
john commented on 25.07.2017 18:34

drop this file into target/linux/ramips/patches-4.9/ on current trunk, ignoring all previous patches. i have had an iperf test run for 2 hours now with near gbit speed using SQM/cake/piece.of.cake setup to rate limit at 600Mbit and have not seen any oopses

camel commented on 25.07.2017 19:13

Pls commit it asao
We will test the trunk. It will be in any case an improvement
Pls let me know if trunk snapshot is compiled.
Then I will test more

Pierre Parent commented on 25.07.2017 21:00

Thanks,

I will try to compile it tomorrow for all boards and I will try to add pptpd

camel commented on 25.07.2017 21:10

Thx
Normally, if the same build modules + same kernel magic as trunk build I can install the packages by myself
I don't think, that it would be a big problem

Pierre Parent commented on 26.07.2017 08:34

Here are the images:

https://www.own-mailbox.com/lede/

@camel: in order to not run out of memory don't download files to /tmp/ during your tests but to /dev/null

Baptiste Jonglez commented on 26.07.2017 08:37

The fix has been pushed to master.

Pierre Parent commented on 26.07.2017 08:39

@Baptiste Jonglez ok great!

(Not sure if it is included in the last build yet though, since it was comited 1 hour ago.)

camel commented on 26.07.2017 08:50

as i can see:
it seems to be, that it already in build, too :) ?

https://git.lede-project.org/?p=source.git;a=summary

and build is from:
Timing:
Start Wed Jul 26 00:13:54 2017
End Wed Jul 26 00:31:31 2017
Elapsed 17 mins, 37 secs

see:
http://phase1.builds.lede-project.org/builders/ramips%2Fmt7621/builds/345

so, we have to wait 1 day (if build commit will be done tonight) longer to use the snapshot trunk :)

I'M very nasty to test - won't wait longer :)

@Biptiste: Thx

dchard commented on 31.07.2017 11:36

@john:

I am testing your fixes (in trunk) for 5 days, and so far there was no crash, warning, or any other indication of the problem. The kernel and system logs are also clean.

How I tested is the following:

1. Run openssl test to load all cores/threads to 100%:

openssl speed md5 sha1 sha256 sha512 des des-ede3 aes-128-cbc aes-192-cbc aes-256-cbc rsa2048 dsa2048 rsa4096 -multi 4

2. In the same time, start Iperf3 as server on the router, and run iperf3 tests on gigabit LAN in paralell with the openssl tests above. This is how I simulated high processor load and network traffic in the same time.

Previously it took only a few minutes to recreate the errors above, now it seems to be gone completely. I will look into to logs every few days to see if anything happens.

Thanks for your hard work!

Pierre Parent commented on 18.09.2017 14:51

Not sure if it is related but today I got with latest version:

[  281.257967] BUG: Bad page map in process hotplug-call  pte:00ffffff pmd:00000000
[  281.267040] page:81022fe0 count:0 mapcount:-1 mapping:  (null) index:0x0[  281.271831] device br-lan entered promiscuous mode
[  281.278495] flags: 0x14(referenced|dirty)
[  281.282487] page dumped because: bad pte
[  281.286439] addr:77e78000 vm_flags:00100077 anon_vma:8e8dd9d8 mapping:8f85ca0c index:12
[  281.294460] file:libgcc_s.so.1 fault:filemap_fault mmap:generic_file_readonly_mmap readpage:squashfs_readpage
[  281.304379] CPU: 3 PID: 21248 Comm: hotplug-call Not tainted 4.9.49 #0
[  281.310880] Stack : 00000000 00000000 80557b2a 0000003a 80406a44 00000000 00000000 80550000
[  281.319232]         8fd988fc 804ebea7 80482a8c 00000003 00005300 80553824 8f85ca0c 804e90c0
[  281.327580]         804e90c0 8006b878 00000000 8006b698 00000000 00000001 80487674 8e869bd4
[  281.335926]         00000000 800a9fe0 00000000 00000000 80557b2a 00000000 8f85ca0c 00869bd4
[  281.344268]         00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  281.352612]         ...
[  281.355051] Call Trace:
[  281.357509] [<8000f7c4>] show_stack+0x54/0x88
[  281.361864] [<801df11c>] dump_stack+0x8c/0xd0
[  281.366213] [<800d0430>] print_bad_pte+0x1d8/0x208
[  281.370983] [<800d2eb0>] unmap_page_range+0x5a4/0x760
[  281.376013] [<800d3398>] unmap_vmas+0x54/0x74
[  281.380358] [<800d861c>] exit_mmap+0x6c/0x138
[  281.384696] [<8002994c>] mmput+0x44/0x120
[  281.388708] [<800f635c>] flush_old_exec+0x59c/0x740
[  281.393587] [<8013c360>] load_elf_binary+0x3d4/0x1174
[  281.398618] [<800f56a0>] search_binary_handler+0xc4/0x25c
[  281.403991] [<800f6f90>] do_execveat_common+0x490/0x600
[  281.409302] Disabling lock debugging due to kernel taint
[  281.414938] BUG: Bad rss-counter state mm:8eaf8700 idx:0 val:-1
[  281.420833] BUG: Bad rss-counter state mm:8eaf8700 idx:1 val:1

After that the router did not respond, even in UART until I rebooted it.

Baptiste Jonglez commented on 18.09.2017 20:26

This looks like an entirely different issue, please open a new bug report.

Loading...

Available keyboard shortcuts

Tasklist

Task Details

Task Editing