LEDE Project

Attached to Project: LEDE Project
Opened by Vittorio Gambaletta (VittGam) - 25.08.2016
Last edited by Felix Fietkau - 17.06.2017

FS#120 - opkg fails with Out of memory on a device with 32 MB of RAM

This happens on a freshly installed image from downloads.lede-project.org for an ar71xx device with 4 MB of flash and 32 MB of RAM (TP-LINK TL-MR13U).

This didn’t happen with Chaos Calmer.

root@lede:~# cat /etc/banner
     _________
    /        /\      _    ___ ___  ___
   /  LE    /  \    | |  | __|   \| __|
  /    DE  /    \   | |__| _|| |) | _|
 /________/  LE  \  |____|___|___/|___|                      lede-project.org
 \        \   DE /
  \    LE  \    /  -----------------------------------------------------------
   \  DE    \  /    Reboot (HEAD, r1439)
    \________\/    -----------------------------------------------------------

root@lede:~# opkg update
[...] (it succeeds)
root@lede:~# opkg install foo
Collected errors:
 * gz_open: fork: Out of memory.
 * pkg_hash_add_from_file: Failed to open /var/opkg-lists/reboot_routing: Out of memory.
root@lede:~# opkg update
Collected errors:
 * gz_open: fork: Out of memory.
 * pkg_hash_add_from_file: Failed to open /var/opkg-lists/reboot_routing: Out of memory.
root@lede:~# free
             total       used       free     shared    buffers     cached
Mem:         27828      15592      12236        668       1028       3352
-/+ buffers/cache:      11212      16616
Swap:            0          0          0
root@lede:~# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/root                 2048      2048         0 100% /rom
tmpfs                    13912       636     13276   5% /tmp
tmpfs                    13912        32     13880   0% /tmp/root
tmpfs                      512         0       512   0% /dev
/dev/mtdblock3             576       212       364  37% /overlay
overlayfs:/overlay         576       212       364  37% /
root@lede:~# 

Cheers,
Vittorio

Closed by  Felix Fietkau
17.06.2017 11:57
Reason for closing:  Fixed
Martin Tippmann commented on 26.08.2016 16:07

See also https://github.com/lede-project/source/issues/237

A workaround that works for me is setting:

OPKG_USE_VFORK=1 opkg install ...

Project Manager
Felix Fietkau commented on 28.08.2016 12:21

fix pushed in r1454

Vittorio Gambaletta (VittGam) commented on 28.08.2016 18:10

The fix seems to work, thanks! But I fail to understand why it's needed now. It didn't fail without vfork on CC with ~5 MB of free RAM, so why does it fail now on LEDE trunk with ~12 MB of free ram?...

Admin
Jo-Philipp Wich commented on 16.09.2016 10:59

We had to revert Felix' original commit due to some issues in opkgs vfork code but I pushed an alternative approach to my staging tree which lets opkg use the external gzip command. As a side effect the opkg binary size is reduced by ~4K.

I would appreciate some testing feedback on http://git.lede-project.org/d692bfec1306a514936eca5805dff6b8fac0ee0e

Pavlo Rudyi commented on 28.09.2016 13:43

The same with TL-741ND, r1657

Workaround:

cat /etc/opkg/distfeeds.conf 
src/gz reboot_core http://downloads.lede-project.org/snapshots/targets/ar71xx/generic/packages
src/gz reboot_base http://downloads.lede-project.org/snapshots/packages/mips_24kc/base
#src/gz reboot_telephony http://downloads.lede-project.org/snapshots/packages/mips_24kc/telephony
src/gz reboot_packages http://downloads.lede-project.org/snapshots/packages/mips_24kc/packages
src/gz reboot_routing http://downloads.lede-project.org/snapshots/packages/mips_24kc/routing
#src/gz reboot_luci http://downloads.lede-project.org/snapshots/packages/mips_24kc/luci
Alberto Bursi commented on 16.10.2016 14:58

@Jo-Philipp Wich

I just tested the two patches on a TL-WR1043ND v1 (32MiB ram, 8 MiB flash), and I'm available for more testing on this device. It is attached with serial, so I can recover anything.

Now installing a single package works fine (before it always failed).

Installing more than one package gives weird errors:

root@lede:/# opkg install luci-app-minidlna
Installing luci-app-minidlna (git-16.288.36935-1e1a706-1) to root...
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/luci/luci-app-minidlna_git-16.288.36935-1e1a706-1_all.ipk.
Installing minidlna (1.1.5-1) to root...
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/packages/minidlna_1.1.5-1_mips_24kc.ipk.
Installing libexif (0.6.21-1) to root...
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/packages/libexif_0.6.21-1_mips_24kc.ipk.
Installing libjpeg (9a-1) to root...
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/packages/libjpeg_9a-1_mips_24kc.ipk.
Installing libsqlite3 (3130000-2) to root...
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/packages/libsqlite3_3130000-2_mips_24kc.ipk.
Installing libffmpeg-mini (2.8.8-2) to root...
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/packages/libffmpeg-mini_2.8.8-2_mips_24kc.ipk.
Configuring libjpeg.
Configuring libexif.
Configuring libsqlite3.
Configuring minidlna.
Collected errors:
 * opkg_install_pkg: Failed to verify the signature of /var/opkg-lists/reboot_packages.
 * opkg_install_cmd: Cannot install package luci-app-minidlna.
 * xsystem: sh: vfork: Out of memory.
 * pkg_run_script: package "libjpeg" postinst script returned status -1.
 * opkg_configure: libjpeg.postinst returned -1.
 * xsystem: sh: vfork: Out of memory.
 * pkg_run_script: package "libexif" postinst script returned status -1.
 * opkg_configure: libexif.postinst returned -1.
 * xsystem: sh: vfork: Out of memory.
 * pkg_run_script: package "libsqlite3" postinst script returned status -1.
 * opkg_configure: libsqlite3.postinst returned -1.
 * xsystem: sh: vfork: Out of memory.
 * pkg_run_script: package "minidlna" postinst script returned status -1.
 * opkg_configure: minidlna.postinst returned -1.

root@lede:/# free
             total       used       free     shared    buffers     cached
Mem:         27820      17964       9856        644        552       4340
-/+ buffers/cache:      13072      14748
Swap:            0          0          0

Manual uninstallation of packages followed, then another try:

Installing luci-app-minidlna (git-16.288.36935-1e1a706-1) to root...
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/luci/luci-app-minidlna_git-16.288.36935-1e1a706-1_all.ipk.
Installing minidlna (1.1.5-1) to root...
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/packages/minidlna_1.1.5-1_mips_24kc.ipk.
Installing libexif (0.6.21-1) to root...
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/packages/libexif_0.6.21-1_mips_24kc.ipk.
Installing libjpeg (9a-1) to root...
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/packages/libjpeg_9a-1_mips_24kc.ipk.
Configuring libexif.
Configuring minidlna.
Error loading shared library libjpeg.so.9: No such file or directory (needed by /usr/bin/minidlna)
Error loading shared library libid3tag.so.0: No such file or directory (needed by /usr/bin/minidlna)
Error loading shared library libsqlite3.so.0: No such file or directory (needed by /usr/bin/minidlna)
Error loading shared library libavformat.so.56: No such file or directory (needed by /usr/bin/minidlna)
Error loading shared library libavutil.so.54: No such file or directory (needed by /usr/bin/minidlna)
Error loading shared library libFLAC.so.8: No such file or directory (needed by /usr/bin/minidlna)
Error loading shared library libogg.so.0: No such file or directory (needed by /usr/bin/minidlna)
Error loading shared library libvorbis.so.0: No such file or directory (needed by /usr/bin/minidlna)
Error relocating /usr/bin/minidlna: ogg_stream_packetpeek: symbol not found
Error relocating /usr/bin/minidlna: ogg_sync_clear: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_total_changes: symbol not found
Error relocating /usr/bin/minidlna: jpeg_resync_to_restart: symbol not found
Error relocating /usr/bin/minidlna: ogg_page_bos: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_busy_timeout: symbol not found
Error relocating /usr/bin/minidlna: ogg_stream_clear: symbol not found
Error relocating /usr/bin/minidlna: jpeg_read_scanlines: symbol not found
Error relocating /usr/bin/minidlna: id3_tag_findframe: symbol not found
Error relocating /usr/bin/minidlna: ogg_page_pageno: symbol not found
Error relocating /usr/bin/minidlna: jpeg_start_decompress: symbol not found
Error relocating /usr/bin/minidlna: jpeg_destroy_compress: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_exec: symbol not found
Error relocating /usr/bin/minidlna: vorbis_info_init: symbol not found
Error relocating /usr/bin/minidlna: id3_ucs4_latin1duplicate: symbol not found
Error relocating /usr/bin/minidlna: id3_field_getstrings: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_open: symbol not found
Error relocating /usr/bin/minidlna: jpeg_write_scanlines: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_libversion_number: symbol not found
Error relocating /usr/bin/minidlna: jpeg_std_error: symbol not found
Error relocating /usr/bin/minidlna: vorbis_comment_init: symbol not found
Error relocating /usr/bin/minidlna: id3_ucs4_utf8duplicate: symbol not found
Error relocating /usr/bin/minidlna: ogg_stream_packetout: symbol not found
Error relocating /usr/bin/minidlna: id3_file_close: symbol not found
Error relocating /usr/bin/minidlna: av_log_set_level: symbol not found
Error relocating /usr/bin/minidlna: ogg_stream_pagein: symbol not found
Error relocating /usr/bin/minidlna: jpeg_set_defaults: symbol not found
Error relocating /usr/bin/minidlna: id3_field_getnstrings: symbol not found
Error relocating /usr/bin/minidlna: id3_file_tag: symbol not found
Error relocating /usr/bin/minidlna: FLAC__metadata_simple_iterator_init: symbol not found
Error relocating /usr/bin/minidlna: ogg_sync_buffer: symbol not found
Error relocating /usr/bin/minidlna: id3_file_open: symbol not found
Error relocating /usr/bin/minidlna: FLAC__metadata_object_delete: symbol not found
Error relocating /usr/bin/minidlna: vorbis_comment_clear: symbol not found
Error relocating /usr/bin/minidlna: FLAC__metadata_simple_iterator_status: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_column_int: symbol not found
Error relocating /usr/bin/minidlna: ogg_sync_wrote: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_errmsg: symbol not found
Error relocating /usr/bin/minidlna: FLAC__metadata_simple_iterator_delete: symbol not found
Error relocating /usr/bin/minidlna: av_register_all: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_get_table: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_column_type: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_step: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_column_int64: symbol not found
Error relocating /usr/bin/minidlna: jpeg_CreateDecompress: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_close: symbol not found
Error relocating /usr/bin/minidlna: ogg_page_granulepos: symbol not found
Error relocating /usr/bin/minidlna: ogg_sync_pageout: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_free_table: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_release_memory: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_snprintf: symbol not found
Error relocating /usr/bin/minidlna: avformat_close_input: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_column_bytes: symbol not found
Error relocating /usr/bin/minidlna: id3_field_getbinarydata: symbol not found
Error relocating /usr/bin/minidlna: jpeg_destroy_decompress: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_column_text: symbol not found
Error relocating /usr/bin/minidlna: jpeg_stdio_src: symbol not found
Error relocating /usr/bin/minidlna: avformat_find_stream_info: symbol not found
Error relocating /usr/bin/minidlna: id3_field_getstring: symbol not found
Error relocating /usr/bin/minidlna: av_dict_get: symbol not found
Error relocating /usr/bin/minidlna: jpeg_read_header: symbol not found
Error relocating /usr/bin/minidlna: jpeg_finish_decompress: symbol not found
Error relocating /usr/bin/minidlna: jpeg_CreateCompress: symbol not found
Error relocating /usr/bin/minidlna: id3_field_getlatin1: symbol not found
Error relocating /usr/bin/minidlna: vorbis_synthesis_headerin: symbol not found
Error relocating /usr/bin/minidlna: av_reduce: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_vmprintf: symbol not found
Error relocating /usr/bin/minidlna: ogg_stream_init: symbol not found
Error relocating /usr/bin/minidlna: id3_field_getfullstring: symbol not found
Error relocating /usr/bin/minidlna: FLAC__metadata_simple_iterator_next: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_prepare: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_threadsafe: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_finalize: symbol not found
Error relocating /usr/bin/minidlna: ogg_page_serialno: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_last_insert_rowid: symbol not found
Error relocating /usr/bin/minidlna: vorbis_info_clear: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_free: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_mprintf: symbol not found
Error relocating /usr/bin/minidlna: jpeg_start_compress: symbol not found
Error relocating /usr/bin/minidlna: ogg_sync_init: symbol not found
Error relocating /usr/bin/minidlna: avformat_open_input: symbol not found
Error relocating /usr/bin/minidlna: FLAC__metadata_simple_iterator_get_block: symbol not found
Error relocating /usr/bin/minidlna: ogg_page_eos: symbol not found
Error relocating /usr/bin/minidlna: jpeg_finish_compress: symbol not found
Error relocating /usr/bin/minidlna: FLAC__metadata_simple_iterator_new: symbol not found
Error relocating /usr/bin/minidlna: av_strerror: symbol not found
Error relocating /usr/bin/minidlna: jpeg_set_quality: symbol not found
Error relocating /usr/bin/minidlna: FLAC__Metadata_SimpleIteratorStatusString: symbol not found
Collected errors:
 * gz_open: fork: Out of memory.
 * opkg_install_pkg: Failed to unpack control files from /tmp/opkg-CeephH/libjpeg_9a-1_mips_24kc.ipk.
 * opkg_install_cmd: Cannot install package luci-app-minidlna.

root@lede:/# free
             total       used       free     shared    buffers     cached
Mem:         27820      18420       9400        972       1112       3940
-/+ buffers/cache:      13368      14452
Swap:            0          0          0

Alberto Bursi commented on 17.10.2016 13:55

tried with swap (it has like 1.5GiB of swap now), installing things still fails hard.
(I'm installing a local package now, as I'm also trying to test other stuff)

root@lede:/mnt# opkg install gdisk_1.0.1-1_mips_24kc.ipk
[ 5094.764231] opkg invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0
[ 5094.771788] CPU: 0 PID: 2312 Comm: opkg Not tainted 4.4.24 #0
[ 5094.777618] Stack : 803e07a4 00000000 00000001 80430000 8198d180 8042ae63 803c1e58 00000908
          8049379c 811c3d1c 00000000 00000003 0006538b 800a7228 00000006 803ca770
          00000000 00000000 803c58c8 811c3bbc 80496542 800a51a4 8042a4d0 00000000
          00000001 801f4700 00000000 00000000 00000000 00000000 00000000 00000000
          00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
          ...
[ 5094.813699] Call Trace:
[ 5094.816181] [<80071b70>] show_stack+0x50/0x84
[ 5094.820577] [<800d3d68>] dump_header.isra.4+0x48/0x130
[ 5094.825768] [<800d414c>] oom_kill_process+0x98/0x43c
[ 5094.830774] [<800d4854>] out_of_memory+0x2e0/0x324
[ 5094.835629] [<800d8154>] __alloc_pages_nodemask+0x684/0x6f0
[ 5094.841312] [<800d314c>] filemap_fault+0x258/0x458
[ 5094.846185] [<800edde0>] __do_fault+0x3c/0xa8
[ 5094.850604] [<800f1084>] handle_mm_fault+0x4e4/0xb84
[ 5094.855635] [<80076820>] __do_page_fault+0x134/0x470
[ 5094.860641] [<80060820>] ret_from_exception+0x0/0x10
[ 5094.865652] 
[ 5094.867151] Mem-Info:
[ 5094.869469] active_anon:1019 inactive_anon:1064 isolated_anon:32
[ 5094.869469]  active_file:0 inactive_file:11 isolated_file:0
[ 5094.869469]  unevictable:0 dirty:0 writeback:1088 unstable:0
[ 5094.869469]  slab_reclaimable:144 slab_unreclaimable:2683
[ 5094.869469]  mapped:1 shmem:2 pagetables:53 bounce:0
[ 5094.869469]  free:256 free_pcp:0 free_cma:0
[ 5094.901444] Normal free:1024kB min:1024kB low:1280kB high:1536kB active_anon:4076kB inactive_anon:4256kB active_file:0kB inactive_file:44kB unevictable:0kB isolated(anon):128kB isolated(file):0kB present:32768kB managed:27820kB mlocked:0kB dirty:0kB writeback:4352kB mapped:4kB shmem:8kB slab_reclaimable:576kB slab_unreclaimable:10732kB kernel_stack:376kB pagetables:212kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:51240 all_unreclaimable? yes
[ 5094.945352] lowmem_reserve[]: 0 0
[ 5094.948710] Normal: 40*4kB (U) 80*8kB (UM) 14*16kB (UM) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1024kB
[ 5094.960562] 1117 total pagecache pages
[ 5094.964345] 1104 pages in swap cache
[ 5094.967942] Swap cache stats: add 4379, delete 3275, find 1170/1704
[ 5094.974256] Free swap  = 1624364kB
[ 5094.977678] Total swap = 1630592kB
[ 5094.981097] 8192 pages RAM
[ 5094.983830] 0 pages HighMem/MovableOnly
[ 5094.987689] 1237 pages reserved
[ 5094.990846] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
[ 5094.999455] [  522]     0   522      295        1       3       0       15             0 ubusd
[ 5095.008139] [  527]     0   527      297        1       3       0       10             0 ash
[ 5095.016644] [  704]     0   704      307        1       4       0       34             0 logd
[ 5095.025237] [  713]     0   713      361        1       4       0       22             0 rpcd
[ 5095.033835] [  782]     0   782      338        5       3       0       18             0 odhcpd
[ 5095.042609] [  823]     0   823      265        1       4       0       10             0 dropbear
[ 5095.051541] [  840]     0   840      380        1       4       0       25             0 uhttpd
[ 5095.060376] [  974]     0   974      297        1       3       0        8             0 ntpd
[ 5095.068971] [ 1859]     0  1859      409        7       4       0       28             0 netifd
[ 5095.077737] [ 1961]     0  1961      230        1       3       0       10             0 odhcp6c
[ 5095.086591] [ 1970]     0  1970      296        1       3       0        8             0 udhcpc
[ 5095.095366] [ 2031]   453  2031      263        1       4       0       16             0 dnsmasq
[ 5095.104226] [ 2306]     0  2306     1871     1001       4       0      633             0 opkg
[ 5095.112821] [ 2312]     0  2312     1668      517       4       0      905             0 opkg
[ 5095.121397] Out of memory: Kill process 2306 (opkg) score 3 or sacrifice child
[ 5095.128687] Killed process 2312 (opkg) total-vm:6672kB, anon-rss:2068kB, file-rss:0kB
Installing gdisk (1.0.1-1) to root...
Collected errors:
 * gz_close: Unzip process killed by signal 9.

another try, this time seems to double-crash.

root@lede:/mnt# opkg install gdisk_1.0.1-1_mips_24kc.ipk
[ 5283.015505] netifd invoked oom-killer: gfp_mask=0x2420848, order=0, oom_score_adj=0
[ 5283.023280] CPU: 0 PID: 1859 Comm: netifd Not tainted 4.4.24 #0
[ 5283.029232] Stack : 803e07a4 00000000 00000001 80430000 8198f980 8042ae63 803c1e58 00000743
          8049379c 80d35a7c 00000000 00000003 0006538b 800a7228 00000006 803ca770
          00000000 00000000 803c58c8 80d3591c 80496542 800a51a4 8042a4d0 00000000
          00000001 801f4700 00000000 00000000 00000000 00000000 00000000 00000000
          00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
          ...
[ 5283.065296] Call Trace:
[ 5283.067778] [<80071b70>] show_stack+0x50/0x84
[ 5283.072175] [<800d3d68>] dump_header.isra.4+0x48/0x130
[ 5283.077366] [<800d414c>] oom_kill_process+0x98/0x43c
[ 5283.082372] [<800d4854>] out_of_memory+0x2e0/0x324
[ 5283.087229] [<800d8154>] __alloc_pages_nodemask+0x684/0x6f0
[ 5283.092888] [<800d12f8>] pagecache_get_page+0x154/0x278
[ 5283.098166] [<80132cbc>] __getblk_slow+0x15c/0x374
[ 5283.103021] [<8015a204>] squashfs_read_data+0x1c8/0x6e8
[ 5283.108290] [<8015a8b8>] squashfs_cache_get+0x194/0x2cc
[ 5283.113577] [<8015e288>] squashfs_readpage_block+0x1a0/0x4d8
[ 5283.119277] [<8015c090>] squashfs_readpage+0x5bc/0x6d0
[ 5283.124487] [<800db8f0>] __do_page_cache_readahead+0x1f8/0x264
[ 5283.130366] [<800d30a0>] filemap_fault+0x1ac/0x458
[ 5283.135231] [<800edde0>] __do_fault+0x3c/0xa8
[ 5283.139626] [<800f1084>] handle_mm_fault+0x4e4/0xb84
[ 5283.144650] [<80076820>] __do_page_fault+0x134/0x470
[ 5283.149657] [<80060820>] ret_from_exception+0x0/0x10
[ 5283.154664] 
[ 5283.156166] Mem-Info:
[ 5283.158485] active_anon:970 inactive_anon:1044 isolated_anon:0
[ 5283.158485]  active_file:30 inactive_file:79 isolated_file:0
[ 5283.158485]  unevictable:0 dirty:0 writeback:1039 unstable:0
[ 5283.158485]  slab_reclaimable:144 slab_unreclaimable:2683
[ 5283.158485]  mapped:1 shmem:0 pagetables:55 bounce:0
[ 5283.158485]  free:256 free_pcp:0 free_cma:0
[ 5283.190374] Normal free:1024kB min:1024kB low:1280kB high:1536kB active_anon:3880kB inactive_anon:4176kB active_file:120kB inactive_file:316kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:32768kB managed:27820kB mlocked:0kB dirty:0kB writeback:4156kB mapped:4kB shmem:0kB slab_reclaimable:576kB slab_unreclaimable:10732kB kernel_stack:376kB pagetables:220kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:51708 all_unreclaimable? yes
[ 5283.234370] lowmem_reserve[]: 0 0
[ 5283.237725] Normal: 72*4kB (UME) 74*8kB (UE) 9*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1024kB
[ 5283.249586] 1161 total pagecache pages
[ 5283.253369] 1052 pages in swap cache
[ 5283.256967] Swap cache stats: add 6510, delete 5458, find 1987/2922
[ 5283.263281] Free swap  = 1624560kB
[ 5283.266702] Total swap = 1630592kB
[ 5283.270120] 8192 pages RAM
[ 5283.272855] 0 pages HighMem/MovableOnly
[ 5283.276713] 1237 pages reserved
[ 5283.279871] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
[ 5283.288479] [  522]     0   522      295        1       3       0       15             0 ubusd
[ 5283.297159] [  527]     0   527      297        1       3       0       10             0 ash
[ 5283.305664] [  704]     0   704      307        1       4       0       34             0 logd
[ 5283.314263] [  713]     0   713      361        1       4       0       22             0 rpcd
[ 5283.322860] [  782]     0   782      338        3       3       0       20             0 odhcpd
[ 5283.331619] [  823]     0   823      265        1       4       0       10             0 dropbear
[ 5283.340568] [  840]     0   840      380        1       4       0       25             0 uhttpd
[ 5283.349335] [  974]     0   974      297        1       3       0        8             0 ntpd
[ 5283.357932] [ 1859]     0  1859      409        5       4       0       30             0 netifd
[ 5283.366701] [ 1961]     0  1961      230        1       3       0       10             0 odhcp6c
[ 5283.375555] [ 1970]     0  1970      296        7       3       0        2             0 udhcpc
[ 5283.384328] [ 2031]   453  2031      263        1       4       0       16             0 dnsmasq
[ 5283.393190] [ 2320]     0  2320     1851      968       5       0      646             0 opkg
[ 5283.401772] [ 2326]     0  2326     1668      492       5       0      930             0 opkg
[ 5283.410357] Out of memory: Kill process 2320 (opkg) score 3 or sacrifice child
[ 5283.417647] Killed process 2326 (opkg) total-vm:6672kB, anon-rss:1968kB, file-rss:0kB
[ 5285.135381] udhcpc invoked oom-killer: gfp_mask=0x2420848, order=0, oom_score_adj=0
[ 5285.143147] CPU: 0 PID: 1970 Comm: udhcpc Not tainted 4.4.24 #0
[ 5285.149098] Stack : 803e07a4 00000000 00000001 80430000 8198ea80 8042ae63 803c1e58 000007b2
          8049379c 81d47ad4 00000000 00000003 0006538b 800a7228 00000006 803ca770
          00000000 00000000 803c58c8 81d47974 80496542 800a51a4 8042a4d0 00000000
          00000001 801f4700 00000000 00000000 00000000 00000000 00000000 00000000
          00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
          ...
[ 5285.185181] Call Trace:
[ 5285.187662] [<80071b70>] show_stack+0x50/0x84
[ 5285.192058] [<800d3d68>] dump_header.isra.4+0x48/0x130
[ 5285.197255] [<800d414c>] oom_kill_process+0x98/0x43c
[ 5285.202257] [<800d4854>] out_of_memory+0x2e0/0x324
[ 5285.207110] [<800d8154>] __alloc_pages_nodemask+0x684/0x6f0
[ 5285.212761] [<800d12f8>] pagecache_get_page+0x154/0x278
[ 5285.218032] [<80132cbc>] __getblk_slow+0x15c/0x374
[ 5285.222898] [<8015a204>] squashfs_read_data+0x1c8/0x6e8
[ 5285.228171] [<8015e414>] squashfs_readpage_block+0x32c/0x4d8
[ 5285.233896] [<8015c090>] squashfs_readpage+0x5bc/0x6d0
[ 5285.239083] [<800db8f0>] __do_page_cache_readahead+0x1f8/0x264
[ 5285.244982] [<800d30a0>] filemap_fault+0x1ac/0x458
[ 5285.249822] [<800edde0>] __do_fault+0x3c/0xa8
[ 5285.254233] [<800f1084>] handle_mm_fault+0x4e4/0xb84
[ 5285.259252] [<80076820>] __do_page_fault+0x134/0x470
[ 5285.264272] [<80060820>] ret_from_exception+0x0/0x10
[ 5285.269266] 
[ 5285.270767] Mem-Info:
[ 5285.273100] active_anon:968 inactive_anon:1046 isolated_anon:0
[ 5285.273100]  active_file:58 inactive_file:64 isolated_file:0
[ 5285.273100]  unevictable:0 dirty:0 writeback:1031 unstable:0
[ 5285.273100]  slab_reclaimable:144 slab_unreclaimable:2685
[ 5285.273100]  mapped:4 shmem:0 pagetables:50 bounce:0
[ 5285.273100]  free:256 free_pcp:0 free_cma:0
[ 5285.305023] Normal free:1024kB min:1024kB low:1280kB high:1536kB active_anon:3872kB inactive_anon:4184kB active_file:232kB inactive_file:256kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:32768kB managed:27820kB mlocked:0kB dirty:0kB writeback:4124kB mapped:16kB shmem:0kB slab_reclaimable:576kB slab_unreclaimable:10740kB kernel_stack:376kB pagetables:200kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:54780 all_unreclaimable? yes
[ 5285.349120] lowmem_reserve[]: 0 0
[ 5285.352473] Normal: 76*4kB (UE) 76*8kB (UME) 7*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1024kB
[ 5285.364334] 1177 total pagecache pages
[ 5285.368108] 1055 pages in swap cache
[ 5285.371706] Swap cache stats: add 6520, delete 5465, find 1988/2927
[ 5285.378019] Free swap  = 1624772kB
[ 5285.381441] Total swap = 1630592kB
[ 5285.384869] 8192 pages RAM
[ 5285.387589] 0 pages HighMem/MovableOnly
[ 5285.391443] 1237 pages reserved
[ 5285.394615] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
[ 5285.403216] [  522]     0   522      295        1       3       0       15             0 ubusd
[ 5285.411888] [  527]     0   527      297        1       3       0       10             0 ash
[ 5285.420395] [  704]     0   704      307        1       4       0       34             0 logd
[ 5285.428991] [  713]     0   713      361        1       4       0       22             0 rpcd
[ 5285.437577] [  782]     0   782      338        4       3       0       19             0 odhcpd
[ 5285.446344] [  823]     0   823      265        1       4       0       10             0 dropbear
[ 5285.455282] [  840]     0   840      380        1       4       0       25             0 uhttpd
[ 5285.464046] [  974]     0   974      297        1       3       0        8             0 ntpd
[ 5285.472624] [ 1859]     0  1859      409        5       4       0       30             0 netifd
[ 5285.481390] [ 1961]     0  1961      230        1       3       0       10             0 odhcp6c
[ 5285.490241] [ 1970]     0  1970      296       10       3       0        2             0 udhcpc
[ 5285.499014] [ 2031]   453  2031      263        1       4       0       16             0 dnsmasq
[ 5285.507875] [ 2320]     0  2320     1851      968       5       0      646             0 opkg
[ 5285.516459] Out of memory: Kill process 2320 (opkg) score 3 or sacrifice child
[ 5285.523748] Killed process 2320 (opkg) total-vm:7404kB, anon-rss:3868kB, file-rss:4kB
Killed
pmgp commented on 21.10.2016 14:02

LEDE head 1949, TPLINK 1043ND v1.
Looks better, now able to list and pipe to grep without going OOM.
Install successful for many packages, but (luci-app-)minidlna is not one of them.

cypa commented on 20.11.2016 15:14

same problem on TP-LINK TL-WR741ND v5 r2244

unable to download

root@k16:/tmp# opkg install libpthread
Installing libpthread (1.1.15-1) to root...
Downloading http://downloads.lede-project.org/snapshots/targets/ar71xx/generic/packages/libpthread_1.1.15-1_mips_24kc.ipk.
Collected errors:
 * xsystem: wget: vfork: Out of memory.
 * opkg_download: Failed to download http://downloads.lede-project.org/snapshots/targets/ar71xx/generic/packages/libpthread_1.1.15-1_mips_24kc.ipk, wget returned -1.
 * opkg_install_pkg: Failed to download libpthread. Perhaps you need to run 'opkg update'?
 * opkg_install_cmd: Cannot install package libpthread.
root@k16:/tmp#

and install any package, even smallest one

root@k16:/tmp# OPKG_USE_VFORK=1 opkg install libpthread_1.1.15-1_mips_24kc.ipk 
Collected errors:
 * gz_open: fork: Out of memory.
 * pkg_init_from_file: Failed to extract control file from libpthread_1.1.15-1_mips_24kc.ipk.
root@k16:/tmp#
cypa commented on 20.11.2016 15:33

commenting reboot_luci and reboot_telephony does the thing!
thanks Pavlo Rudyi

Alberto Bursi commented on 03.12.2016 16:48

@Jo-Philipp Wich
@cypa

I think the only way to solve this is to simply don't keep local package lists on "low mem" devices. Each time opkg needs to look at package lists it downloads, reads and deletes each package list singularly.

It will have a pretty big latency (as commands like "opkg list" will have to download and parse all lists before giving an output), but would preserve functionality.

Since opkg is a package manager, I think list reading performance can be sacrificed here.

Project Manager
Felix Fietkau commented on 27.12.2016 09:22

Please test the latest version

Alberto Bursi commented on 31.12.2016 18:57

Still fails when it really shouldn't. it seems wget fails to download a 4kib package while I have 8 MiB of free RAM + 12 MiB of caches.

root@lede:/# opkg install luci-app-minidlna
Installing luci-app-minidlna (git-16.365.41161-ca79611-1) to root...
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/luci/luci-app-minidlna_git-16.365.41161-ca79611-1_all.ipk.
Collected errors:
* xsystem: wget: vfork: Out of memory.
* opkg_download: Failed to download http://downloads.lede-project.org/snapshots/packages/mips_24kc/luci/luci-app-minidlna_git-16.365.41161-ca79611-1_all.ipk, wget returned -1.
* opkg_install_pkg: Failed to download luci-app-minidlna. Perhaps you need to run 'opkg update'?
* opkg_install_cmd: Cannot install package luci-app-minidlna.

root@lede:/# free

           total       used       free     shared    buffers     cached

Mem: 27940 19156 8784 560 1044 3704
-/+ buffers/cache: 14408 13532
Swap: 0 0 0

root@lede:/# cat /proc/meminfo
MemTotal: 27940 kB
MemFree: 7996 kB
MemAvailable: 10660 kB
Buffers: 1120 kB
Cached: 3716 kB
SwapCached: 0 kB
Active: 3840 kB
Inactive: 2172 kB
Active(anon): 1704 kB
Inactive(anon): 32 kB
Active(file): 2136 kB
Inactive(file): 2140 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 1184 kB
Mapped: 1644 kB
Shmem: 560 kB
Slab: 7068 kB
SReclaimable: 1892 kB
SUnreclaim: 5176 kB
KernelStack: 328 kB
PageTables: 216 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 13968 kB
Committed_AS: 4312 kB
VmallocTotal: 1048372 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB

Sreekanth commented on 02.01.2017 18:41

Doesn't work for me too.


BusyBox v1.25.1 () built-in shell (ash)

     _________
    /        /\      _    ___ ___  ___
   /  LE    /  \    | |  | __|   \| __|
  /    DE  /    \   | |__| _|| |) | _|
 /________/  LE  \  |____|___|___/|___|                      lede-project.org
 \        \   DE /
  \    LE  \    /  -----------------------------------------------------------
   \  DE    \  /    Reboot (SNAPSHOT, r2709-b7677f0)
    \________\/    -----------------------------------------------------------

root@lede:~# free
             total       used       free     shared    buffers     cached
Mem:         27808      20792       7016        868       1408       5884
-/+ buffers/cache:      13500      14308
Swap:            0          0          0
root@lede:~# opkg update
Downloading http://downloads.lede-project.org/snapshots/targets/ar71xx/generic/packages/Packages.gz.
Downloading http://downloads.lede-project.org/snapshots/targets/ar71xx/generic/packages/Packages.sig.
Signature check failed.
Remove wrong Signature file.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/base/Packages.gz.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/base/Packages.sig.
Signature check failed.
Remove wrong Signature file.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/luci/Packages.gz.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/luci/Packages.sig.
Signature check failed.
Remove wrong Signature file.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/packages/Packages.gz.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/packages/Packages.sig.
Signature check failed.
Remove wrong Signature file.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/routing/Packages.gz.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/routing/Packages.sig.
Signature check failed.
Remove wrong Signature file.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/telephony/Packages.gz.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/telephony/Packages.sig.
Signature check failed.
Remove wrong Signature file.
Collected errors:
 * xsystem: wget: vfork: Out of memory.
 * opkg_download: Failed to download http://downloads.lede-project.org/snapshots/targets/ar71xx/generic/packages/Packages.gz, wget returned -1.
 * xsystem: wget: vfork: Out of memory.
 * opkg_download: Failed to download http://downloads.lede-project.org/snapshots/targets/ar71xx/generic/packages/Packages.sig, wget returned -1.
 * xsystem: wget: vfork: Out of memory.
 * opkg_download: Failed to download http://downloads.lede-project.org/snapshots/packages/mips_24kc/base/Packages.gz, wget returned -1.
 * xsystem: wget: vfork: Out of memory.
 * opkg_download: Failed to download http://downloads.lede-project.org/snapshots/packages/mips_24kc/base/Packages.sig, wget returned -1.
 * xsystem: wget: vfork: Out of memory.
 * opkg_download: Failed to download http://downloads.lede-project.org/snapshots/packages/mips_24kc/luci/Packages.gz, wget returned -1.
 * xsystem: wget: vfork: Out of memory.
 * opkg_download: Failed to download http://downloads.lede-project.org/snapshots/packages/mips_24kc/luci/Packages.sig, wget returned -1.
 * xsystem: wget: vfork: Out of memory.
 * opkg_download: Failed to download http://downloads.lede-project.org/snapshots/packages/mips_24kc/packages/Packages.gz, wget returned -1.
 * xsystem: wget: vfork: Out of memory.
 * opkg_download: Failed to download http://downloads.lede-project.org/snapshots/packages/mips_24kc/packages/Packages.sig, wget returned -1.
 * xsystem: wget: vfork: Out of memory.
 * opkg_download: Failed to download http://downloads.lede-project.org/snapshots/packages/mips_24kc/routing/Packages.gz, wget returned -1.
 * xsystem: wget: vfork: Out of memory.
 * opkg_download: Failed to download http://downloads.lede-project.org/snapshots/packages/mips_24kc/routing/Packages.sig, wget returned -1.
 * xsystem: wget: vfork: Out of memory.
 * opkg_download: Failed to download http://downloads.lede-project.org/snapshots/packages/mips_24kc/telephony/Packages.gz, wget returned -1.
 * xsystem: wget: vfork: Out of memory.
 * opkg_download: Failed to download http://downloads.lede-project.org/snapshots/packages/mips_24kc/telephony/Packages.sig, wget returned -1.
root@lede:~#
root@lede:~# free
             total       used       free     shared    buffers     cached
Mem:         27808      19292       8516        392       1408       4116
-/+ buffers/cache:      13768      14040
Swap:            0          0          0
root@lede:~#
root@lede:~#

On a second attempt, opkg update was successful.

root@lede:~# opkg update
Downloading http://downloads.lede-project.org/snapshots/targets/ar71xx/generic/packages/Packages.gz.
Updated list of available packages in /var/opkg-lists/reboot_core.
Downloading http://downloads.lede-project.org/snapshots/targets/ar71xx/generic/packages/Packages.sig.
Signature check passed.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/base/Packages.gz.
Updated list of available packages in /var/opkg-lists/reboot_base.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/base/Packages.sig.
Signature check passed.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/luci/Packages.gz.
Updated list of available packages in /var/opkg-lists/reboot_luci.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/luci/Packages.sig.
Signature check passed.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/packages/Packages.gz.
Updated list of available packages in /var/opkg-lists/reboot_packages.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/packages/Packages.sig.
Signature check passed.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/routing/Packages.gz.
Updated list of available packages in /var/opkg-lists/reboot_routing.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/routing/Packages.sig.
Signature check passed.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/telephony/Packages.gz.
Updated list of available packages in /var/opkg-lists/reboot_telephony.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/telephony/Packages.sig.
Signature check passed.

However, when I tried to install a package, then it failed again.

root@lede:~# opkg install luci-app-adblock
Installing luci-app-adblock (git-17.002.34912-a48c5d2-1) to root...
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/luci/luci-app-adblock_git-17.002.34912-a48c5d2-1_all.ipk.
Collected errors:
 * xsystem: wget: vfork: Out of memory.
 * opkg_download: Failed to download http://downloads.lede-project.org/snapshots/packages/mips_24kc/luci/luci-app-adblock_git-17.002.34912-a48c5d2-1_all.ipk, wget returned -1.
 * opkg_install_pkg: Failed to download luci-app-adblock. Perhaps you need to run 'opkg update'?
 * opkg_install_cmd: Cannot install package luci-app-adblock.
Christian Hamar commented on 05.02.2017 16:52

Hey there! Got this problem too with a ver 1.X (4MB / 32MB) tplink 1043ND router.

On some forums there was a suggestion for opkg vfork* mem error.

sysctl -w vm.min_free_kbytes=0

when i used this and run opkg update then Packages list updated. I can see a couple of packages via opkg list-upgradable .

But. I can upgrade small packages, such as odhcpd or odhcpd6 or luci-proto-ppp *

But when i want to run opkg upgrade luci-base or luci-theme-bootstrap* (eg. bigger than ~10kb) then it just stuck at downloading.

root@OpenWrt:/tmp# opkg upgrade luci-mod-admin-full
Upgrading luci-mod-admin-full on root from git-17.025.85178-472dc4b-1 to git-17.033.24085-e306ee6-1...
Downloading http://downloads.lede-project.org/releases/17.01.0-rc1/packages/mips_24kc/luci/luci-mod-admin-full_git-17.033.24085-e306ee6-1_mips_24kc.ipk.

When i go to /tmp and do an ls -la then i see that there is an opkg-XYJkfds tmp dir for this package. I check that. I got the .ipk file already downloaded for full size and a temporary directory. Probably that should be where .ipk extracted.

Nah thats empty. Router working fine i can terminate opkg only via killing the process.

btw

root@OpenWrt:/tmp# df -h
Filesystem Size Used Available Use% Mounted on
/dev/root 2.3M 2.3M 0 100% /rom
tmpfs 13.8M 988.0K 12.8M 7% /tmp
/dev/mtdblock3 4.4M 1.5M 2.9M 34% /overlay
overlayfs:/overlay 4.4M 1.5M 2.9M 34% /
tmpfs 512.0K 0 512.0K 0% /dev

What information i can provide to the team about this "bug"? :)

(btw, with owrt 15.0X version i did not have this problem on this router, so wondering)

eripek commented on 14.02.2017 20:21

@C. Hamar: there is no TL-WR1043NDv1 I'd know of, that has 4MB flash memory. They all have 8.

I'd suggest using image builder and add zram and zram-swap, ...

unless dev team decides to activate that as a default option with uci set system.@system[0].zram_size_mb=24 (or higher) for this build target.

That is the most quick and not so dirty solution for this some memory issue at the cost of loosing some cpu cycles...

The main problem of the memory issue is sysupgrade failing during flash. Be sure to have the most recent bootloader for easy (tftp) recovery of your device! Especially elder devices have a UBoot dated back to 2009. It makes sense to flash back an image of the original firmware (follow revert procedures for NOboot/stripped original firmware w/o the bootloader) and then flash the newest original firmware (with "_boot_" in it's filename) available all over again using the TP-Link orginal firmware webinterface. After that a Uboot version from 2014 supporting reset-button-triggered TFTP should be installed on the router.

Murilo Pontes commented on 18.02.2017 01:52

Hi guys,

I have a TL-WR1043NDv1 running Reboot (17.01.0-rc2, r3131-42f3c1f)
Bizarre "sysctl -w vm.min_free_kbytes=0" help but it not work everytime.
With Openwrt 15.05.1 opkg run fine.

root@LEDE:~# opkg install tcpdump
Installing tcpdump (4.9.0-1) to root...
Downloading http://downloads.lede-project.org/releases/17.01.0-rc2/packages/mips_24kc/base/tcpdump_4.9.0-1_mips_24kc.ipk * xsystem: wget: vfork: Out of memory.
* opkg_download: Failed to download http://downloads.lede-project.org/releases/17.01.0-rc2/packages/mips_24kc/base/tcpdump_4.9.0-1_mips_24kc.ipk, wget returned -1.

Admin
Jo-Philipp Wich commented on 19.02.2017 18:21

I pushed an opkg update to master with https://git.lede-project.org/b65dc04 which will make opkg install processes consume much less memory.

Rod Egan commented on 21.02.2017 06:32

Using the 17.01.0 built code on a tplink wr842n v1 (32M ram) and a wr842m v3 (64M ram)
LEDE Reboot 17.01.0 r3205-59508e3 / LuCI lede-17.01 branch (git-17.051.53299-a100738)

without the opkg update from Jo-Phillip above, ie unpatched code, opkg fails on v1 device 32M ram, works ok on 64M ram device. So the issue is still present in the 17.01 build.

Admin
Jo-Philipp Wich commented on 21.02.2017 10:43

Yes, the opkg update didn't make it to 17.01 as it was a too severe change.

Rod Egan commented on 21.02.2017 11:42

pity as it pretty much makes 17.01 useless for anyone with a 32M ram device who wants to add any packages. May need a warning on the release notes about the constraint.
I'll look at trunk as an option.

Admin
Jo-Philipp Wich commented on 21.02.2017 12:48

I know. We'll plan to ship the opkg update with 17.01.1 which is due mid/end of March so not all is lost :)

The opkg rework just happened too late for .0

Admin
Jo-Philipp Wich commented on 21.02.2017 12:49

It should be possible to just install opkg from master on a 17.01 system.

Rod Egan commented on 21.02.2017 13:13

Appreciate the time & effort spent in sorting this all out.
Happy to search but, how do I install opkg from master to an existing install?

Admin
Jo-Philipp Wich commented on 21.02.2017 13:21

E.g. on an ar71xx device it would be

opkg install http://downloads.lede-project.org/snapshots/targets/ar71xx/generic/packages/opkg_2017-02-20-bec9804c-1_mips_24kc.ipk
Alberto Bursi commented on 21.02.2017 13:33

@jow opkg fails to do that because of same low ram issues, at least here in a test system still on older snapshot.

Lol waiting for March is going to be a blood bath.
I would really think it's worht it to make an exception for opkg and get it in release regardless.

Rod Egan commented on 21.02.2017 13:57

Running that update then 'opkg update' has changed all the feeds to 'snapshot' which then raises all sorts of errors when trying to run opkg.
You need to manually alter the opkg feeds back to 17.01 from snapshot to get things running as expected.
It also seems to cause any software package view via luci to fail. ie no packages are shown in the 'available tab'.
It is possible to run opkg from the command line.

Lajos commented on 23.02.2017 13:40

How about temporary shut down all services except what needed for opkg to run?
I am so sad as I have some devices with 32MB of RAM.

evgeneys commented on 28.02.2017 13:28

Final release 17.01 on Tl-841 v5 :(
collected errors:
* xsystem: wget: vfork: Out of memory.
* opkg_download: Failed to download http://downloads.lede-project.org/releases/17.01.0/packages/mips_24kc/routing/miniupnpd_2.0.20161216-1_mips_24kc.ipk, wget returned -1.
* opkg_install_pkg: Failed to download miniupnpd. Perhaps you need to run 'opkg update'?
* opkg_install_cmd: Cannot install package miniupnpd.

Alberto Bursi commented on 28.02.2017 19:38

See a few posts above, the fix for this will be in LEDE 17.01.1 which is due mid/end of March.

Snotto commented on 03.03.2017 12:22

Hey guys,

i used zram to solve my problems with my four wr1043n v1.
Maybe you should try this ...

Lajos commented on 03.03.2017 16:23

@Snotto: Could you please provide more details how to solve it with zram.

Snotto commented on 05.03.2017 11:58

Simply install both zram packages you could find with opkg.
After you instlled it opkg update and install will be fine.

kmod-zram and zram-swap are the packages.

But take care ... until now all my 32 MB devices get problems over time with these
17.01. LEDE. I went back to CC 15.01. because its rock solid !

Rob White commented on 18.03.2017 11:58

I have been getting round this using imagebuilder.
As Alberto says above "See a few posts above, the fix for this will be in LEDE 17.01.1 which is due mid/end of March. "
Is this confirmed?

If not I will have to revert back to OpenWrt CC for our latest production rollout.

Rob White commented on 18.03.2017 12:45

Further to my comment of 18.03.2017 11:58, I have downloaded the opkg version in today's snapshot (opkg_2017-03-15-cfe46c7d-1_mips_24kc.ipk) and installed in 17.01.0 and it works fine.

Magician24 commented on 01.04.2017 03:26

Also been affected by this with a 32MB RAM router, but been able to get around it by disabling some package lists.

Any updates on the release of 17.01.1 with this fix? It was said earlier "mid/end" of this month but still nothing so far.

Alberto Bursi commented on 09.04.2017 16:26

LEDE 17.01.1 is being prepared and should be released by the Easter holidays, according to this announcement http://lists.infradead.org/pipermail/lede-dev/2017-April/007003.html

Murilo Pontes commented on 17.06.2017 02:34

running opkg with LEDE 17.01.2 in WR1043v1 (32mb ram/8mb flash) is fine for me. I installed several packages without any errors.

Rob White commented on 17.06.2017 10:31

This was fixed in 17.01.1. Glad to see it is still fixed in .2 :D

muhaha commented on 09.07.2017 17:04

Not working 17.01.2

TP-Link TL-WR1043N/ND v1
LEDE Reboot 17.01.2 r3435-65eec8bd5f / LuCI lede-17.01 branch (git-17.163.57307-c79bb96)

root@LEDE:~# free
             total       used       free     shared    buffers     cached
Mem:         28176      23472       4704       1324       1388       5668
-/+ buffers/cache:      16416      11760
Swap:            0          0          0
root@LEDE:~# df -h |grep overlay
/dev/sda1                 3.7G    266.1M      3.4G   7% /overlay
overlayfs:/overlay        3.7G    266.1M      3.4G   7% /
Rob White commented on 09.07.2017 21:54

@muhaha
Can you be more specific with what is not working. I have had complete success on various hardware with less free memory than you are showing.

muhaha commented on 10.07.2017 06:45

@Rob White,
still getting Out of memory

Here is stacktrace

Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.480381] sh invoked oom-killer: gfp_mask=0x2420848, order=0, oom_score_adj=0
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.487839] CPU: 0 PID: 3520 Comm: sh Not tainted 4.4.71 #0
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.493464] Stack : 80387554 00000000 00000001 803e0000 00000000 00000000 00000000 00000000
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.493464] 	  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.493464] 	  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.493464] 	  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.493464] 	  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.493464] 	  ...
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.529579] Call Trace:[<80071a50>] 0x80071a50
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.534126] [<80071a50>] 0x80071a50
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.537641] [<800d24ac>] 0x800d24ac
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.541179] [<8009f02c>] 0x8009f02c
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.544712] [<800d2890>] 0x800d2890
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.548234] [<8009f974>] 0x8009f974
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.551766] [<8017ef9c>] 0x8017ef9c
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.555302] [<800d2f98>] 0x800d2f98
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.558817] [<800d6898>] 0x800d6898
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.562365] [<800cf9ec>] 0x800cf9ec
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.565890] [<80131484>] 0x80131484
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.569423] [<80131704>] 0x80131704
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.572996] [<80131954>] 0x80131954
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.576535] [<80158ba0>] 0x80158ba0
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.580099] [<8015cdb0>] 0x8015cdb0
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.583646] [<801598a4>] 0x801598a4
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.587187] [<8015aa2c>] 0x8015aa2c
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.590761] [<800da034>] 0x800da034
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.594295] [<800cebbc>] 0x800cebbc
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.597888] [<800d17e4>] 0x800d17e4
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.601417] [<800ef350>] 0x800ef350
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.604939] [<8011a4cc>] 0x8011a4cc
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.608448] [<801af9e0>] 0x801af9e0
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.612010] [<800ec5e4>] 0x800ec5e4
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.615521] [<800ef350>] 0x800ef350
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.619041] [<800ef888>] 0x800ef888
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.622583] [<80082188>] 0x80082188
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.626108] [<80081b90>] 0x80081b90
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.629653] [<8007670c>] 0x8007670c
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.633217] [<8009f02c>] 0x8009f02c
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.636727] [<80083408>] 0x80083408
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.640247] [<8009f974>] 0x8009f974
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.643784] [<8009b3f8>] 0x8009b3f8
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.647302] [<80066c10>] 0x80066c10
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.650843] [<80060820>] 0x80060820
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.654362] [<80065060>] 0x80065060
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.657874] 
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.659376] Mem-Info:
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.661710] active_anon:3065 inactive_anon:22 isolated_anon:0
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.661710]  active_file:165 inactive_file:172 isolated_file:0
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.661710]  unevictable:0 dirty:1 writeback:0 unstable:0
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.661710]  slab_reclaimable:248 slab_unreclaimable:1253
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.661710]  mapped:33 shmem:300 pagetables:97 bounce:0
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.661710]  free:354 free_pcp:0 free_cma:0
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.693684] Normal free:1416kB min:1024kB low:1280kB high:1536kB active_anon:12260kB inactive_anon:88kB active_file:660kB inactive_file:688kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:32768kB managed:28176kB mlocked:0kB dirty:4kB writeback:0kB mapped:132kB shmem:1200kB slab_reclaimable:992kB slab_unreclaimable:5012kB kernel_stack:448kB pagetables:388kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scMon Jul 10 08:41:43 2017 kern.warn kernel: [40390.737592] lowmem_reserve[]: 0 0
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.740969] Normal: 268*4kB (UME) 31*8kB (UM) 6*16kB (UM) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1416kB
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.753004] 639 total pagecache pages
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.756697] 0 pages in swap cache
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.760037] Swap cache stats: add 0, delete 0, find 0/0
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.765305] Free swap  = 0kB
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.768202] Total swap = 0kB
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.771113] 8192 pages RAM
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.773835] 0 pages HighMem/MovableOnly
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.777689] 1148 pages reserved
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.780863] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.789462] [  504]     0   504      295       15       3       0        0             0 ubusd
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.798150] [  505]     0   505      224       10       3       0        0             0 askfirst
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.807090] [  704]     0   704      313       41       3       0        0             0 logd
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.815685] [  713]     0   713      361       23       4       0        0             0 rpcd
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.824283] [  760]     0   760      426       41       4       0        0             0 netifd
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.833055] [  786]     0   786      355       37       5       0        0             0 odhcpd
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.841828] [  839]     0   839      265       11       3       0        0             0 dropbear
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.850774] [  857]     0   857      988      130       4       0        0             0 uhttpd
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.859532] [  872] 65534   872      846       76       4       0        0             0 openvpn
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.868389] [ 1001]     0  1001      297        9       3       0        0             0 udhcpc
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.877155] [ 1003]     0  1003      255       13       3       0        0             0 odhcp6c
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.886012] [ 1128]     0  1128      298       10       4       0        0             0 ntpd
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.894610] [ 1534]     0  1534      345       61       3       0        0             0 dynamic_dns_upd
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.904168] [ 1650]     0  1650      422       31       3       0        0             0 hostapd
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.913027] [ 2226]   453  2226      562      316       4       0        0             0 dnsmasq
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.921886] [ 2406]     0  2406      298       10       3       0        0             0 sleep
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.930571] [ 2444]     0  2444      282       29       4       0        0             0 dropbear
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.939502] [ 3456]     0  3456      732      233       3       0        0             0 luci
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.948098] [ 3486]     0  3486      297       10       4       0        0             0 sh
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.956516] [ 3487]     0  3487     1569     1312       4       0        0             0 opkg
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.965111] [ 3490]     0  3490      649      165       4       0        0             0 luci
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.973710] [ 3491]     0  3491      649      164       4       0        0             0 luci
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.982307] [ 3509]     0  3509      297       10       3       0        0             0 sh
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.990731] [ 3510]     0  3510      791       47       4       0        0             0 wget
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.999313] [ 3511]     0  3511      297       10       3       0        0             0 grep
Mon Jul 10 08:41:43 2017 kern.info kernel: [40391.007910] [ 3520]     0  3520      298       11       4       0        0             0 sh
Mon Jul 10 08:41:43 2017 kern.err kernel: [40391.016318] Out of memory: Kill process 3487 (opkg) score 181 or sacrifice child
Mon Jul 10 08:41:43 2017 kern.err kernel: [40391.023786] Killed process 3487 (opkg) total-vm:6276kB, anon-rss:5224kB, file-rss:24kB
Mon Jul 10 08:41:44 2017 daemon.err uhttpd[857]: Killed
root@LEDE:~# opkg update
Downloading http://downloads.lede-project.org/releases/17.01.2/targets/ar71xx/generic/packages/Packages.gz
*** Failed to download the package list from http://downloads.lede-project.org/releases/17.01.2/targets/ar71xx/generic/packages/Packages.gz

Downloading http://downloads.lede-project.org/releases/17.01.2/packages/mips_24kc/base/Packages.gz
*** Failed to download the package list from http://downloads.lede-project.org/releases/17.01.2/packages/mips_24kc/base/Packages.gz

Downloading http://downloads.lede-project.org/releases/17.01.2/packages/mips_24kc/luci/Packages.gz
*** Failed to download the package list from http://downloads.lede-project.org/releases/17.01.2/packages/mips_24kc/luci/Packages.gz

Downloading http://downloads.lede-project.org/releases/17.01.2/packages/mips_24kc/packages/Packages.gz
*** Failed to download the package list from http://downloads.lede-project.org/releases/17.01.2/packages/mips_24kc/packages/Packages.gz

Downloading http://downloads.lede-project.org/releases/17.01.2/packages/mips_24kc/routing/Packages.gz
*** Failed to download the package list from http://downloads.lede-project.org/releases/17.01.2/packages/mips_24kc/routing/Packages.gz

Collected errors:
 * xsystem: wget: vfork: Out of memory.
 * opkg_download: Failed to download http://downloads.lede-project.org/releases/17.01.2/targets/ar71xx/generic/packages/Packages.gz, wget returned -1.
 * xsystem: wget: vfork: Out of memory.
 * opkg_download: Failed to download http://downloads.lede-project.org/releases/17.01.2/packages/mips_24kc/base/Packages.gz, wget returned -1.
 * xsystem: wget: vfork: Out of memory.
 * opkg_download: Failed to download http://downloads.lede-project.org/releases/17.01.2/packages/mips_24kc/luci/Packages.gz, wget returned -1.
 * xsystem: wget: vfork: Out of memory.
 * opkg_download: Failed to download http://downloads.lede-project.org/releases/17.01.2/packages/mips_24kc/packages/Packages.gz, wget returned -1.
 * xsystem: wget: vfork: Out of memory.
 * opkg_download: Failed to download http://downloads.lede-project.org/releases/17.01.2/packages/mips_24kc/routing/Packages.gz, wget returned -1.
root@LEDE:~# free
             total       used       free     shared    buffers     cached
Mem:         28176      20732       7444       1200        560       3896
-/+ buffers/cache:      16276      11900
Swap:            0          0          0
root@LEDE:~# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 2.3M      2.3M         0 100% /rom
tmpfs                    13.8M      1.2M     12.6M   9% /tmp
/dev/sda1                 3.7G    266.2M      3.4G   7% /overlay
overlayfs:/overlay        3.7G    266.2M      3.4G   7% /
tmpfs                   512.0K         0    512.0K   0% /dev

EDIT:
No luck with opkg_2017-05-03-04e279eb-1_mips_24kc.ipk

Rob White commented on 11.07.2017 08:40

@muhaha
I just downloaded the current release and re-flashed a brand new Ubiquiti airGateway. It works perfectly. As you can see from the following, I seem to have less free memory than you do and I have not installed anything or have anything extra running. I suggest you try a new re-flash to check nothing is corrupted.

\    LE  \    /  -----------------------------------------------------------
 \  DE    \  /    Reboot (17.01.2, r3435-65eec8bd5f)
  \________\/    -----------------------------------------------------------

root@LEDE:~# free

           total       used       free     shared    buffers     cached

Mem: 28176 23748 4428 1116 2060 8284
-/+ buffers/cache: 13404 14772
Swap: 0 0 0
root@LEDE:~# df -h
Filesystem Size Used Available Use% Mounted on
/dev/root 2.3M 2.3M 0 100% /rom
tmpfs 13.8M 1.0M 12.7M 8% /tmp
tmpfs 13.8M 48.0K 13.7M 0% /tmp/root
tmpfs 512.0K 0 512.0K 0% /dev
/dev/mtdblock5 3.9M 280.0K 3.6M 7% /overlay
overlayfs:/overlay 3.9M 280.0K 3.6M 7% /
root@LEDE:~#

Rob White commented on 11.07.2017 09:07

@muhaha
I think I see your problem. On booting up I get:
root@LEDE:~# opkg update
Downloading http://downloads.lede-project.org/releases/17.01.2/targets/ar71xx/generic/packages/Packages.gz Updated list of available packages in /var/opkg-lists/reboot_core
Downloading http://downloads.lede-project.org/releases/17.01.2/targets/ar71xx/generic/packages/Packages.sig Signature check passed.
Downloading http://down.... etc

I then loaded up php7 and ran a script that used up loads of memory, taking free down to less than 1MB. Then I get your problem. There is obviously still a threshold where opkg runs out of memory. Try a reboot and see if it helps ;)

Rob White commented on 11.07.2017 09:23

@cypa
@Jo-Philipp Wich
Maybe there is still a small issue here, but I can work round it 100% by a reboot.
Perhaps getting opkg to check free first and if not enough suggest stopping some services and/or rebooting?...

Admin
Jo-Philipp Wich commented on 12.08.2017 12:23

Well, if a system is out of memory, it is out of memory. No amount of opkg optimizations will change that. I lowered the resource usage of opkg by 60-70% percent, thats as far as it will get.

If you still suffer from out of memory conditions then your environment simply cannot accomodate opkg anymore. Either try to reduce the amounts of feeds or consider precompiling images yourself.

Loading...

Available keyboard shortcuts

Tasklist

Task Details

Task Editing