Mass Deleting With split and map

-
# First off - this is how I got into the mess in the first place.

 # Step 1: make a really big random file:
head -c $(( 1024 * 1024 * 1024 )) /dev/urandom > $(mktemp '/DataSwap/big.XXXXXXXX')

# Step 2: Screw up and split it into a million (actually 1024*1024)
# separate 1024byte files.
split -b 1024 /DataSwap/big.KFGgJs8S '/DataSwap/parts'

 # Trying to delete them normally just fails because the command line is two long.

 # This seems to be about as fast as I can get, using a whole
# bunch of parallel deleters.

# First make separate files of 1000 entries to remove and put in shared
# memory for speed (linuxisum).
ls | split - '/dev/shm/lses' 

 # Now make a function which can read a block and delete all
# the files listed.
function rm_block { for f in $(cat $1); do rm "/DataSwap/$f"; done; }

# A wrapper to easily put that in the background.
function rm_block_bg { rm_block $1 & }

# Now kick off the delete:
ls /dev/shm/ls* | map rm_block_bg

# Go make a cup of tea whilst the linux kernel schedules
# up the work for you...

Comments

Post a Comment

Popular posts from this blog

Bithon: Run Python Interactively Inside Bash

-
cat > bithon.sh << EOF # This script allows python to run as child of bash # attached to anonymous pipes and all commands in # bash starting py are sent to python.   function launch_python {   # Close any existing use of these fds.   exec 3>&-   exec 4<&-   pyin=$(mktemp -u)   pyot=$(mktemp -u)   mkfifo $pyin   mkfifo $pyot   python -u -i <$pyin &>$pyot &   pypid=$!   exec 3> $pyin   exec 4< $pyot   # Make the pipes 'anonymous'.   rm $pyin   rm $pyot   echo print >&3   while read -u 4 line   do     if [ "${line}" == '>>>' ]     then       break     fi     echo  "# ${line}"   done   echo 'import sys;sys.ps2=""' >&3 } function py_read {   local line   while read -u 4 line   do     if [ "${line}" == '...
Read more

oche, lik echo but a bit easier to use.

-
function oche { printf "%s\n" "$@"; } Alexanders-MacBook-Pro:~ alexanderturner$ oche "a b c" a b c Alexanders-MacBook-Pro:~ alexanderturner$ oche a b c a b c Alexanders-MacBook-Pro:~ alexanderturner$ thing=$(ls -l) Alexanders-MacBook-Pro:~ alexanderturner$ oche "$thing" total 98888 -rw-r--r--     1 alexanderturner  staff      1286 20 Mar  2016 #.emacs# drwxr-xr-x     4 alexanderturner  staff       136 23 Dec  2014 AWS drwxr-xr-x     6 alexanderturner  staff       204 22 May  2014 Accounts drwx------     4 alexanderturner  staff       136 25 Oct  2015 Applications   ...
Read more

Parsing Columns From Files WITHOUT awk

-
# awk is cool - but sometimes jumping from bash to awk to bash gets clunky. # We don't actually have use awk - we can just leverage bash internal parsing. # This is a classic awk example but now we have the really useful function map # which lets us do the same time in bash. function size_name {     print $5 $9 } function map {     local l;     while read -r l; do         $1 $l;     done } ls -l SonicField/src/cpp/lib/ | map size_name | column -t 168       build.sh 2108      stream.hpp 15676048  stream.hpp_out # Let's try this as a one liner: _tmp () { print $5 $9; }; ls -l SonicField/src/cpp/lib/ | map _tmp  | column -t # awk is simpler but not by much if we assume map is part of your utils ls -l SonicField/src/cpp/lib/ | awk '{print $5, $9}'  | column -t 168       build.sh 2...
Read more