cat abuse with split

-
# cat gets used all the time just to pipe the contents of a
# file to stdout.
# But it is actually for concatenating files.
# It has a partner in crime called split.
# Working together they are very powerful for parallel processing.
# split - do work in parallel - cat

 # split then cat will produce out output file which is identical to the input:

# Make a 1 gig file of random bytes on my external ssd.
time head -c $(( 1024 * 1024 * 1024 )) /dev/urandom > $(mktemp '/DataSwap/big.XXXXXXXX')

real    0m5.738s
user    0m0.057s
sys    0m5.680s

# Yey - that was fast!
[aturner@Alexanders-MBP ~]$ split -b $(( 1024 * 1024)) /DataSwap/big.* '/DataSwap/parts'
[aturner@Alexanders-MBP ~]$ ls /DataSwap/parts*
/DataSwap/partsaa  /DataSwap/partsgp  /DataSwap/partsne  /DataSwap/partstt    /DataSwap/partszabi  /DataSwap/partszahx
/DataSwap/partsab  /DataSwap/partsgq  /DataSwap/partsnf  /DataSwap/partstu    /DataSwap/partszabj  /DataSwap/partszahy
/DataSwap/partsac  /DataSwap/partsgr  /DataSwap/partsng  /DataSwap/partstv    /DataSwap/partszabk  /DataSwap/partszahz
# ... stuff not shown as there are 1024 files!

# cat them back together - note how split creates file names
# such that ls automatically creates the correct ordering.
time cat $(ls /DataSwap/parts*) > /DataSwap/Rebuild

real    0m1.949s
user    0m0.017s
sys    0m0.802s
 
# Again - time is not too shabby (over 500 mega bytes per second)!
cmp /DataSwap/big.oTFSbZnD /DataSwap/Rebuild 
[[ $? == 0 ]] && print Yey
Yey
 

Comments

Post a Comment

Popular posts from this blog

Bithon: Run Python Interactively Inside Bash

-
cat > bithon.sh << EOF # This script allows python to run as child of bash # attached to anonymous pipes and all commands in # bash starting py are sent to python.   function launch_python {   # Close any existing use of these fds.   exec 3>&-   exec 4<&-   pyin=$(mktemp -u)   pyot=$(mktemp -u)   mkfifo $pyin   mkfifo $pyot   python -u -i <$pyin &>$pyot &   pypid=$!   exec 3> $pyin   exec 4< $pyot   # Make the pipes 'anonymous'.   rm $pyin   rm $pyot   echo print >&3   while read -u 4 line   do     if [ "${line}" == '>>>' ]     then       break     fi     echo  "# ${line}"   done   echo 'import sys;sys.ps2=""' >&3 } function py_read {   local line   while read -u 4 line   do     if [ "${line}" == '.' ]     then       break     fi     if [ "${line}" != '>>>' ]     then        line="${line#>>> }"   
Read more

Parsing Columns From Files WITHOUT awk

-
# awk is cool - but sometimes jumping from bash to awk to bash gets clunky. # We don't actually have use awk - we can just leverage bash internal parsing. # This is a classic awk example but now we have the really useful function map # which lets us do the same time in bash. function size_name {     print $5 $9 } function map {     local l;     while read -r l; do         $1 $l;     done } ls -l SonicField/src/cpp/lib/ | map size_name | column -t 168       build.sh 2108      stream.hpp 15676048  stream.hpp_out # Let's try this as a one liner: _tmp () { print $5 $9; }; ls -l SonicField/src/cpp/lib/ | map _tmp  | column -t # awk is simpler but not by much if we assume map is part of your utils ls -l SonicField/src/cpp/lib/ | awk '{print $5, $9}'  | column -t 168       build.sh 2108      stream.hpp 15676048  stream.hpp_out # But remember that awk is a separate process space so you do not  # have access to the bash state in the same way. For example:
Read more

oche, lik echo but a bit easier to use.

-
function oche { printf "%s\n" "$@"; } Alexanders-MacBook-Pro:~ alexanderturner$ oche "a b c" a b c Alexanders-MacBook-Pro:~ alexanderturner$ oche a b c a b c Alexanders-MacBook-Pro:~ alexanderturner$ thing=$(ls -l) Alexanders-MacBook-Pro:~ alexanderturner$ oche "$thing" total 98888 -rw-r--r--     1 alexanderturner  staff      1286 20 Mar  2016 #.emacs# drwxr-xr-x     4 alexanderturner  staff       136 23 Dec  2014 AWS drwxr-xr-x     6 alexanderturner  staff       204 22 May  2014 Accounts drwx------     4 alexanderturner  staff       136 25 Oct  2015 Applications   ...
Read more