<div dir="ltr">This is a great resource. Thanks for sharing.</div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Sep 25, 2014 at 9:23 PM, <span dir="ltr"><<a href="mailto:jep200404@columbus.rr.com" target="_blank">jep200404@columbus.rr.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On Wed, 03 Sep 2014 20:54:48 -0400, Scott Merrill <<a href="mailto:skippy@skippy.net">skippy@skippy.net</a>> wrote:<br>
<br>
> Jim Prior will present on editing the command line and scripting<br>
> on the command line. Jim will show one or more ways of doing<br>
> something, and then others will share better ways of do that same<br>
> thing. This should be an interactive and pretty practical<br>
> meeting.<br>
><br>
> for f in *.htm; do echo mv -i "$f" "${f%%.htm}.html";done<br>
> for f in index.*.html; do echo mv -i "$f" "ndx.${f##index.}";done<br>
<br>
Release early, release often.<br>
Fun was had playing around with the command line.<br>
<br>
This presentation was inspired by mailing list thread[1], which said in part:<br>
<br>
> I am no longer shocked at how many do not understand the<br>
> basic productivity enhancement tools available to them like<br>
<br>
> shell history<br>
> screen<br>
> commands like watch<br>
> in place shell programming (basic for loops, etc)<br>
<br>
and<br>
<br>
> And _most_ only know about the arrow keys for history.<br>
<br>
Of those, this presentation was about:<br>
o searching the history<br>
o editing the command line<br>
o loops<br>
o some long strings of programs piped together<br>
<br>
First a little presentation about giving presentations.<br>
<br>
The key thing is that the presentation should be easily viewed by everyone<br>
in the audience, including those in the back row.<br>
<br>
Resolution<br>
<br>
24 * 80 is about max resolution<br>
<br>
Colors<br>
<br>
Room lights on projection screens is common and bad, so maximize<br>
the contrast in your content to salvage what contrast is available<br>
to make it easier to see.<br>
<br>
GOOD:<br>
<br>
white on black<br>
black on white<br>
<br>
TOLERABLE:<br>
<br>
pastels on white<br>
<br>
BAD, VERY VERY BAD:<br>
<br>
colorized text on black<br>
<br>
combining multiple commands, especially loops<br>
In the following "stgfopl" means something that generates filenames, one per<br>
line. Examples could include:<br>
<br>
ls *<br>
ls *.JPG<br>
find . -type f -name '*.pdf' -print<br>
<br>
loops<br>
<br>
Here are some common patterns I use:<br>
<br>
for i in `seq 1234 4566`;do echo do something with "$i";done<br>
for f in *.pdf; do echo do something with "$f";done<br>
stgfopl | while read f; do echo do something with "$f";done<br>
stgfopl | xargs some command<br>
<br>
Most of the rest of this presentation will play with examples of those<br>
common patterns.<br>
<br>
I used to have loops like:<br>
<br>
i=5;while [ $i -le 10 ]; do echo $i;i=`expr $i + 1`;done<br>
<br>
I now usually use seq instead:<br>
<br>
for i in `seq 5 10`;do echo $i;done<br>
<br>
The former style still has a place where the output of seq is too much<br>
for the shell, of if the iteration is not just simple arithmetic<br>
sequence. For example, consider the following:<br>
<br>
i=1;while [ $i -le `echo '2^32' | bc` ]; do echo $i;i=`expr $i + $i`;done<br>
<br>
or for following links, such as in a chain of soft links, or a<br>
linked listg.<br>
<br>
find . -name '*.pdf' -print | xargs wc<br>
find . -name '*.pdf' -print | while read f; do echo do something with "$f";done<br>
<br>
for i in `seq 2622 2635`; do mv -i /media/jep/disk/DCIM/100GEDSC/GEDC`printf '%04d' "$i"`.* .;done<br>
for i in `seq 2622 2635`; do f=GEDC`printf '%04d' "$i"`.JPG;mv -i "$f" "${f%%.JPG}.jpeg" ;done<br>
<br>
Study, compare, and contrast:<br>
<br>
"${f%%.JPG}"<br>
"${f%.JPG}"<br>
"${f##hello.}"<br>
"${f#hello}"<br>
<br>
Some loops go on forever:<br>
<br>
while true; do time factor 1111111111111111111;done #32-bit<br>
while true; do time factor 11111111111111111111111111111111111111;done #64-bit<br>
while true; do date | tr '\n' '\r';sleep .1;done<br>
watch -n .1 "date | tr '\n' '\r'"<br>
<br>
history | sed -e 's/^ *[0-9][0-9]* [0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9] //gi' | sort | uniq -c | sort -nr | head<br>
<br>
Use echo to see what commands would be like (i.e., dry run) before actually<br>
doing commands.<br>
<br>
for f in *.JPG; do echo mv -i "$f" "${f%%.JPG}.jpeg" ;done<br>
for f in *.JPG; do mv -i "$f" "${f%%.JPG}.jpeg" ;done<br>
<br>
classic pipelines<br>
<br>
# Find the most common commands.<br>
# Often there are multiple correct ways to do something.<br>
history | awk '{print $4}' | sort | uniq -c | sort -n | tail<br>
history | awk '{print $4}' | sort | uniq -c | sort -n | tac | head<br>
history | awk '{print $4}' | sort | uniq -c | sort -nr | head<br>
<br>
but the focus of this presentation is more complicated stuff,<br>
particularly loops.<br>
<br>
How long of a line is too long?<br>
<br>
If it makes your head hurt, it's too long.<br>
If it slows you down, it's too long.<br>
Different people tolerate different amounts of complexity.<br>
If you're repeating some monster command often,<br>
it might be a good candidate to put into a shell script.<br>
<br>
history options<br>
<br>
The history can have timestamps. This can help one look for an old command that<br>
one ran some time ago, by narrowing how much of the history one looks at. When<br>
developing scripts, I will often paste multiple lines from them at once. The<br>
command timestamps can document how fast things worked. Hence the following<br>
line in .bashrc.<br>
<br>
HISTTIMEFORMAT="%F %T "<br>
<br>
# Comment commands<br>
<br>
Temporarily parking a command<br>
<br>
In the middle of writing some command, I realize that I need to run<br>
some other command first, so I put '#' at beginning of line and execute<br>
it, so the history remembers the command I was working on. Then run the<br>
command I needed to do first, then go back in recent history to get the<br>
abandoned command, delete the leading # and execute it.<br>
<br>
to document the history<br>
<br>
Russ mentioned deliberating executing a comment command, so that it<br>
ends up in the history to document the following (or preceding)<br>
commands.<br>
<br>
new bash exploit<br>
<br>
<a href="http://lists.colug.net/pipermail/colug-432/2014-September/003356.html" target="_blank">http://lists.colug.net/pipermail/colug-432/2014-September/003356.html</a><br>
<a href="http://fedoramagazine.org/flaw-discovered-in-the-bash-shell-update-your-fedora-systems/" target="_blank">http://fedoramagazine.org/flaw-discovered-in-the-bash-shell-update-your-fedora-systems/</a><br>
<a href="http://www.zdnet.com/unixlinux-bash-critical-security-hole-uncovered-7000034021/" target="_blank">http://www.zdnet.com/unixlinux-bash-critical-security-hole-uncovered-7000034021/</a><br>
<br>
alias lll='ls -l --full-time'<br>
<br>
The --full-time output is easier for downstream programs to sort by date<br>
and/or time. Avoids difficulty of sorting month abbreviations. Also, the<br>
format does not switch for files more than a year old (time of day versus<br>
year). Compare the output of ll versus lll:<br>
<br>
jep@presentation:~$ ll -tr /etc<br>
total 1276<br>
-rw-r--r-- 1 root root 2064 Nov 23 2006 netscsid.conf<br>
...<br>
-rw-r--r-- 1 root root 1189 Sep 25 20:20 mtab<br>
jep@presentation:~$ lll -tr /etc<br>
total 1276<br>
-rw-r--r-- 1 root root 2064 2006-11-23 14:33:10.000000000 -0500 netscsid.conf<br>
...<br>
-rw-r--r-- 1 root root 1189 2014-09-25 20:20:46.194351828 -0400 mtab<br>
jep@presentation:~$<br>
<br>
some nifty things are available in bash, but not Bourne shell<br>
avoiding them is good for portability<br>
<br>
source instead of . (from csh, thanks Rob)<br>
<br>
$(command) instead of `command`<br>
<br>
$(command) can be nested.<br>
`command` can not be nested<br>
<br>
>(command)<br>
<br>
... | tee "$f" | tee >(md5sum) >(sha1sum) >(sha256sum) /dev/null<br>
... | tee "$f" >(gzip >"$f".gz) >(bzip2 >"$f".bz2) /dev/null<br>
drive=/dev/sr0;readrawcd $drive | tee >(md5sum) >(sha1sum) >(sha256sum) >/dev/null;sleep 10;eject $drive;date<br>
drive=xubuntu-12.04.4-desktop-i386.iso;readrawcd $drive | tee >(md5sum) >(sha1sum) >(sha256sum) >/dev/null;sleep 10;eject $drive;date<br>
<br>
tees tease: for fun try the above tees without the /dev/null<br>
<br>
<(command)<br>
<br>
vidf old new<br>
<br>
jep@presentation:~$ cat ~/bin/vidf<br>
# vim -O "$1" "$2" <(diff "$1" "$2")<br>
vim -O <(diff "$1" "$2") "$1" "$2"<br>
jep@presentation:~$<br>
<br>
$((arithmetic expression)) (e.g., $(($i + 1))<br>
<br>
i=5<br>
echo `expr $i + 1`<br>
echo `echo "$i + 1" | bc`<br>
echo `echo "print $i + 1" | python`<br>
# How to do above with awk instead of python?<br>
<br>
Miscellaneous<br>
<br>
Begin dangerous commands with '#' or echo while editing them.<br>
Remove the '#' or echo after one is satisfied that the command is correct.<br>
<br>
<br>
# rm -rf / foo<br>
# rm -rf /foo<br>
rm -rf /foo<br>
<br>
echo rm -rf / foo<br>
echo rm -rf /foo<br>
rm -rf /foo<br>
<br>
sudo !! is handy for rerunning command with sudo<br>
<br>
rm -rf /tmp/*<br>
sudo !!<br>
<br>
cmatrix text "screensave"<br>
<br>
On a sacrificial Ubuntu box in an xterm:<br>
<br>
sudo apt-get update<br>
sudo apt-get install cmatrix<br>
cmatrix<br>
<br>
.bashrc<br>
<br>
Many programs pay attention to the following:<br>
<br>
EDITOR=vim<br>
export EDITOR<br>
<br>
history options: I commented out some of Xubuntu's defaults.<br>
<br>
# don't put duplicate lines or lines starting with space in the history.<br>
# See bash(1) for more options<br>
#HISTCONTROL=ignoreboth<br>
<br>
# append to the history file, don't overwrite it<br>
shopt -s histappend<br>
<br>
Jason and Russ, please elaborate on the benefits and drawbacks of<br>
"shopt -s histappend".<br>
<br>
# for setting history length see HISTSIZE and HISTFILESIZE in bash(1)<br>
HISTSIZE=400000<br>
HISTFILESIZE=400000<br>
# HISTCONTROL=ignoredups:ignorespace<br>
<br>
I want all commands in the history. I _do_ want to save the<br>
duplicate commands. Sometimes repeated commands are necessary for<br>
some task. Having the history remember the repeated commands is<br>
helpful for repeating the whole sequence again later. Sometimes I<br>
want to see how long each command took. Timestamps for repeated<br>
commands answer that. My command to show the most common commands<br>
relies upon those duplicate commands being in the<br>
history.<br>
<br>
unset HISTCONTROL<br>
HISTFILESIZE=400000<br>
HISTSIZE=400000<br>
HISTTIMEFORMAT="%F %T "<br>
export HISTCONTROL<br>
export HISTFILESIZE<br>
export HISTSIZE<br>
<br>
EMACS is default for editing command line and history.<br>
set -o vi to select vi for editing command line<br>
<br>
git stuff<br>
<br>
I need to use git version control more. Good candidates are:<br>
<br>
gas price stuff<br>
.bashrc<br>
.vimrc<br>
vidf<br>
<br>
I found <a href="http://www.git-scm.com/book/en/Git-on-the-Server" target="_blank">http://www.git-scm.com/book/en/Git-on-the-Server</a> and other git<br>
tutorials on that site to be helpful. Rebasing is cool.<br>
<br>
locate foo | xargs md5sum | sort<br>
<br>
Russ, what the line number and URL for your nasty example from<br>
<a href="https://github.com/herrold/" target="_blank">https://github.com/herrold/</a>?<br>
<br>
Is Apple's default search engine now duckduckgo, not Google anymore?<br>
Has Apple's canary disappeared?<br>
<br>
tmux<br>
<br>
Jason, please elaborate on virtue of using ^Q.<br>
<br>
tmux pipe-pane -o 'cat >>'$PWD/tmux.output.`date +%Y%m%d-%H%M%S`<br>
<br>
Screenkey was used to show what keys were typed to edit the command line and<br>
history. Need to tweak screenkey to:<br>
<br>
put it out of the way to waste less space on screen<br>
make it smaller<br>
make it persist (a log file would be nice too)<br>
<br>
<a href="https://launchpad.net/screenkey??" target="_blank">https://launchpad.net/screenkey??</a>??<br>
<br>
I would be happy with key logger output showing in an xterm window.<br>
<br>
# Redirect stdout and stderr with timestamping to log file.<br>
exec 1> >(timestamper | tee -a log) 2>&1<br>
<br>
jep@presentation:~$ cat /home/jep/tools/timestamper.c<br>
#include <stdlib.h><br>
#include <stdio.h><br>
#include <unistd.h><br>
#include <time.h><br>
#include <sys/time.h><br>
#include <string.h><br>
<br>
#define False (0)<br>
#define True (!False)<br>
<br>
typedef unsigned char flag;<br>
<br>
#define ArrayLength(x) (sizeof(x)/sizeof(*(x)))<br>
<br>
void print_timestamp(void)<br>
{<br>
char buf[80];<br>
int i;<br>
struct timeval now;<br>
struct tm *b;<br>
<br>
gettimeofday(&now,NULL);<br>
b=localtime(&now.tv_sec);<br>
i=sprintf(buf,"%04d-%02d-%02d %02d:%02d:%02d.%06d "<br>
, b->tm_year+1900<br>
, b->tm_mon+1<br>
, b->tm_mday<br>
, b->tm_hour<br>
, b->tm_min<br>
, b->tm_sec<br>
, (unsigned int)now.tv_usec);<br>
<br>
write(STDOUT_FILENO,buf,i);<br>
}<br>
<br>
int main(int argc,char *argv[])<br>
{<br>
unsigned char buf[1];<br>
flag please_print_timestamp=True;<br>
int i;<br>
<br>
while ((i=read(STDIN_FILENO,buf,ArrayLength(buf)))>0) {<br>
if (please_print_timestamp) {<br>
please_print_timestamp=False;<br>
print_timestamp();<br>
}<br>
write(STDOUT_FILENO,buf,i);<br>
if (buf[0]=='\n')<br>
please_print_timestamp=True;<br>
}<br>
<br>
return EXIT_SUCCESS;<br>
}<br>
jep@presentation:~/monarch/dev-code$<br>
<br>
Impedance matching:<br>
<br>
Cracking a whip reminds me of how tsunamis can travel in fast, short, long<br>
waves over vast stretches of deep water, then become tall waves in short<br>
water. What's the electrical analogy to cracking whips and tsunamis? It's<br>
probably a bunch of capacitors dumping into some tuned LC network. Hmmm.<br>
Study microwave strip stuff. It also reminds me of Francis turbines.<br>
<br>
wp:is a prefix for wikipedia<br>
wp:Gödel, Escher, Bach<br>
<br>
[1] It got interesting, starting around here:<br>
<a href="http://lists.colug.net/pipermail/colug-432/2014-July/003223.html" target="_blank">http://lists.colug.net/pipermail/colug-432/2014-July/003223.html</a><br>
<a href="http://lists.colug.net/pipermail/colug-432/2014-July/003225.html" target="_blank">http://lists.colug.net/pipermail/colug-432/2014-July/003225.html</a><br>
<a href="http://lists.colug.net/pipermail/colug-432/2014-July/thread.html#3223" target="_blank">http://lists.colug.net/pipermail/colug-432/2014-July/thread.html#3223</a><br>
<br>
_______________________________________________<br>
colug-432 mailing list<br>
<a href="mailto:colug-432@colug.net">colug-432@colug.net</a><br>
<a href="http://lists.colug.net/mailman/listinfo/colug-432" target="_blank">http://lists.colug.net/mailman/listinfo/colug-432</a><br>
</blockquote></div><br></div>