mostlyobvious

Disk Usage and File Indexing

7 min read

Disk Usage (no index)

dua-cli

https://github.com/Byron/dua-cli

  • defaults to cli output but has a tui in i mode
  • probably the winner on speed

gdu

https://github.com/dundee/gdu

  • about as fast as dua
  • more options than dua

dust

https://github.com/bootandy/dust

  • non-interactive tree summary view

ncdu

https://dev.yorhel.nl/ncdu

Baobab

GUI Disk Usage Analyzer https://wiki.gnome.org/action/show/Apps/DiskUsageAnalyzer?action=show&redirect=Apps%2FBaobab

Duplicate Files, with Indexes

duc

duc https://duc.zevv.nl/ https://github.com/zevv/duc/ http://manpages.ubuntu.com/manpages/bionic/man1/duc.1.html

  • yay -S duc-git
  • Generates an index

dupd

http://www.virkki.com/dupd/ https://github.com/jvirkki/dupd

Local File Search

Diskover Community Edition

https://github.com/diskoverdata/diskover-community

  • file system indexer into elasticsearch, with web interface

FESS

https://fess.codelibs.org/ https://github.com/codelibs/docker-fess/pkgs/container/fess

Performance Comparison

See: https://www.reddit.com/r/commandline/comments/sekd0w/ncdu_ncurses_disk_usage_see_which_directories_and/

There were conflicting reports on performance so I ran a few of my own tests on two separate machines and a mix of media and file systems. dua (2.18.0) and gdu (5.20.0) were both many times (4-8X) faster than ncdu (2.2.1). duc (1.4.5) runs at about the same speed as ncdu but it’s worth noting that it writes the results of the index into a DB for future use.

ext4 386GB, 3M files on Sabrent Rocket 4.0 2TB NVM.e PCIe4.0, Ryzen 9 5950X on X570

❯ hyperfine --export-markdown=bench-cold.md --prepare 'sync; echo 3 | sudo tee /proc/sys/vm/drop_caches' \ 'gdu -npc ~' 'gdu -gnpc ~' 'dua ~' 'duc index ~' 'ncdu -0 -o /dev/null ~'
Benchmark 1:  gdu -npc ~
  Time (mean ± σ):     37.718 s ±  6.831 s    [User: 10.804 s, System: 18.125 s]
  Range (min … max):   30.487 s … 50.842 s    10 runs
 
Benchmark 2: gdu -gnpc ~
  Time (mean ± σ):     27.721 s ±  0.442 s    [User: 27.063 s, System: 17.883 s]
  Range (min … max):   27.040 s … 28.618 s    10 runs
 
Benchmark 3: dua ~
  Time (mean ± σ):     17.265 s ±  0.517 s    [User: 7.673 s, System: 27.435 s]
  Range (min … max):   16.171 s … 17.984 s    10 runs
 
Benchmark 4: duc index ~
  Time (mean ± σ):     75.887 s ± 10.573 s    [User: 3.363 s, System: 10.801 s]
  Range (min … max):   70.686 s … 105.849 s    10 runs
 
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet PC without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
 
Benchmark 5: ncdu -0 -o /dev/null ~
  Time (mean ± σ):     71.605 s ±  1.030 s    [User: 1.944 s, System: 9.409 s]
  Range (min … max):   69.870 s … 73.216 s    10 runs
 
Summary
  'dua ~' ran
    1.61 ± 0.05 times faster than 'gdu -gnpc ~'
    2.18 ± 0.40 times faster than ' gdu -npc ~'
    4.15 ± 0.14 times faster than 'ncdu -0 -o /dev/null ~'
    4.40 ± 0.63 times faster than 'duc index ~'

I also added a separate dust run, it looks to slot in between dua and gdu for performance:

hyperfine --export-markdown=bench-cold.md --prepare 'sync; echo 3 | sudo tee /proc/sys/vm/drop_caches' \ 'dust ~'
Benchmark 1:  dust ~
  Time (mean ± σ):     21.716 s ±  1.199 s    [User: 5.219 s, System: 16.820 s]
  Range (min … max):   19.926 s … 23.866 s    10 runs

ext4 3.4TB, 2.1M files on Sabrent Rocket 4.0 Plus NVM.e PCIe4.0, Ryzen 7 3700X on X570

hyperfine --export-markdown=bench-cold.md --prepare 'sync; echo 3 | sudo tee /proc/sys/vm/drop_caches' \ 'gdu -npc /data/fast4tb' 'gdu -gnpc /data/fast4tb' 'dua /data/fast4tb' 'duc index /data/fast4tb' 'ncdu -0 -o /dev/null /data/fast4tb'
Benchmark 1:  gdu -npc /data/fast4tb
  Time (mean ± σ):      3.436 s ±  0.402 s    [User: 5.391 s, System: 16.341 s]
  Range (min … max):    2.997 s …  4.244 s    10 runs
 
Benchmark 2: gdu -gnpc /data/fast4tb
  Time (mean ± σ):      3.001 s ±  0.070 s    [User: 8.499 s, System: 16.801 s]
  Range (min … max):    2.895 s …  3.087 s    10 runs
 
Benchmark 3: dua /data/fast4tb
  Time (mean ± σ):      3.112 s ±  0.029 s    [User: 3.051 s, System: 17.161 s]
  Range (min … max):    3.081 s …  3.170 s    10 runs
 
Benchmark 4: duc index /data/fast4tb
  Time (mean ± σ):     20.735 s ±  0.051 s    [User: 1.637 s, System: 8.116 s]
  Range (min … max):   20.641 s … 20.817 s    10 runs
 
Benchmark 5: ncdu -0 -o /dev/null /data/fast4tb
  Time (mean ± σ):     21.155 s ±  0.037 s    [User: 1.371 s, System: 7.528 s]
  Range (min … max):   21.084 s … 21.212 s    10 runs
 
Summary
  'gdu -gnpc /data/fast4tb' ran
    1.04 ± 0.03 times faster than 'dua /data/fast4tb'
    1.15 ± 0.14 times faster than ' gdu -npc /data/fast4tb'
    6.91 ± 0.16 times faster than 'duc index /data/fast4tb'
    7.05 ± 0.16 times faster than 'ncdu -0 -o /dev/null /data/fast4tb'

ext4 7.4TB, 300K files on Samsung SSD 870 QVO 8TB SATA 6Gb/s, Ryzen 7 3700X on X570

lhl@nas /d/media> hyperfine --export-markdown=bench-cold.md --prepare 'sync; echo 3 | sudo tee /proc/sys/vm/drop_caches' \ 'gdu -npc /data/media' 'gdu -gnpc /data/media' 'dua /data/media' 'duc index /data/media' 'ncdu -0 -o /dev/null /data/media'
Benchmark 1:  gdu -npc /data/media
  Time (mean ± σ):     783.1 ms ±   5.4 ms    [User: 905.2 ms, System: 2158.8 ms]
  Range (min … max):   776.1 ms … 790.1 ms    10 runs
 
Benchmark 2: gdu -gnpc /data/media
  Time (mean ± σ):     883.9 ms ± 218.8 ms    [User: 1525.9 ms, System: 2143.1 ms]
  Range (min … max):   777.2 ms … 1432.1 ms    10 runs
 
  Warning: The first benchmarking run for this command was significantly slower than the rest (1.116 s). This could be caused by (filesystem) caches that were not filled until after the first run. You should consider using the '--warmup' option to fill those caches before the actual benchmark. Alternatively, use the '--prepare' option to clear the caches before each timing run.
 
Benchmark 3: dua /data/media
  Time (mean ± σ):     790.5 ms ±   2.3 ms    [User: 521.1 ms, System: 2234.7 ms]
  Range (min … max):   786.3 ms … 793.8 ms    10 runs
 
Benchmark 4: duc index /data/media
  Time (mean ± σ):      6.035 s ±  0.017 s    [User: 0.187 s, System: 1.153 s]
  Range (min … max):    6.019 s …  6.066 s    10 runs
 
Benchmark 5: ncdu -0 -o /dev/null /data/media
  Time (mean ± σ):      6.019 s ±  0.028 s    [User: 0.173 s, System: 1.062 s]
  Range (min … max):    5.957 s …  6.051 s    10 runs
 
Summary
  ' gdu -npc /data/media' ran
    1.01 ± 0.01 times faster than 'dua /data/media'
    1.13 ± 0.28 times faster than 'gdu -gnpc /data/media'
    7.69 ± 0.06 times faster than 'ncdu -0 -o /dev/null /data/media'
    7.71 ± 0.06 times faster than 'duc index /data/media'

zfs 17GB, 206K files on 6 x 4TB Crucial MX500 SATA SSD (20TB RAIDZ), Ryzen 7 3700X on X570

hyperfine --export-markdown=bench-cold.md --prepare 'sync; echo 3 | sudo tee /proc/sys/vm/drop_caches' \ 'gdu -npc /data/z/Zotero' 'gdu -gnpc /data/z/Zotero' 'dua /data/z/Zotero' 'duc index /data/z/Zotero' 'ncdu -0 -o /dev/null /data/z/Zotero'

Benchmark 1:  gdu -npc /data/z/Zotero
  Time (mean ± σ):     371.3 ms ±   5.9 ms    [User: 596.5 ms, System: 4010.1 ms]
  Range (min … max):   362.3 ms … 381.3 ms    10 runs
 
Benchmark 2: gdu -gnpc /data/z/Zotero
  Time (mean ± σ):     387.1 ms ±   6.0 ms    [User: 985.6 ms, System: 3851.6 ms]
  Range (min … max):   376.4 ms … 397.1 ms    10 runs
 
Benchmark 3: dua /data/z/Zotero
  Time (mean ± σ):     332.4 ms ±   8.3 ms    [User: 397.0 ms, System: 4013.2 ms]
  Range (min … max):   319.5 ms … 343.7 ms    10 runs
 
Benchmark 4: duc index /data/z/Zotero
  Time (mean ± σ):      1.795 s ±  0.013 s    [User: 0.123 s, System: 1.683 s]
  Range (min … max):    1.776 s …  1.816 s    10 runs
 
Benchmark 5: ncdu -0 -o /dev/null /data/z/Zotero
  Time (mean ± σ):      1.754 s ±  0.004 s    [User: 0.114 s, System: 1.623 s]
  Range (min … max):    1.748 s …  1.758 s    10 runs
 
Summary
  'dua /data/z/Zotero' ran
    1.12 ± 0.03 times faster than ' gdu -npc /data/z/Zotero'
    1.16 ± 0.03 times faster than 'gdu -gnpc /data/z/Zotero'
    5.28 ± 0.13 times faster than 'ncdu -0 -o /dev/null /data/z/Zotero'
    5.40 ± 0.14 times faster than 'duc index /data/z/Zotero'

Discussion

2022-12-07 Discussion of ncdu and related tools https://news.ycombinator.com/item?id=33893815

2018-09-17 Comparison of duplicate file finder performance http://www.virkki.com/jyri/articles/index.php/duplicate-finder-performance-2018-edition/

2018-08-01 My discussion on fdupes and friends https://randomfoo.net/2018/08/01/classic-code

2015-02-19 Comparison of duplicate file finder speeds http://elipapa.github.io/blog/fastest-way-to-find-dups.html

Old

sist2 https://github.com/simon987/sist2 https://dataarchivist.net/posts/sist2/

  • Easy to index
  • Easy to send into Elasticsearch
  • Dies on web - no idea why

FESS https://fess.codelibs.org/

  • Manual Install (zip)
  • Manually run Elasticsearch (zip)
  • Managed to get everything running, but index dies very early on…

To Try:

Want