Bash Scripting

So here I was, all set to start getting all of my notes into one place on bash scripting, when I came across an obscure note in my pile of paper. Something I had actually scribbled in there some time ago, but either a) forgot I wrote it there, and as such never referenced it, or b) glanced down at the note and couldn't immediately make out what it said. See, some of my note taking occurs at 3AM, or thereabouts, under the influence of a tremendous amount of coffee and tea, making my notes a bit un readable by most humans in this particular plane of existance.

The original note said: LDP - bash programming howto.
This was crossed out in green crayon, and written directly below was: LDP - ABS.

Intuitive.

LDP - bash programming howto: http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO.html
LDP - ABS (Advanced Bash Scripting) : http://tldp.org/LDP/abs/html/index.html

The latter actually has everything that is in my notes, with the exception of my notes pertaining to whatever problem I was attempting to solve at the time, and the examples in teh ABS are generic, and thus "More gooderer"(tm) than my notes.

Why did I bother writing all this drivel if my intent was to merely point the reader to those two resources? Only so I could use the term "Giant Monkey Robot Ninja Pirate Viking" in a sentence. And also, so there was a place for me to go look in the event I ever need to find these resources again.

Update to "The Best Hotdogs on Planet Earth"

The original blog post here: http://lifeoncastro.blogspot.com/2008/01/best-hot-dogs-on-planet-earth.html

So, a couple of folks wanted to know exactly where it was - my childhood hot dog stand, the ultimate in what makes a dog a dog.. hot dog mecca.


View Larger Map

There it is - in all of it's glory. If you have business in Newark, or live in the tri-state area (really, it's that worth it!) go. For the sake of your soul, go.

Plus - you can pick up an "Eat Me" t-shirt while you are there. Oh.. and you should also fedex me a dozen or so of those bad boys.

Linux Interview - Just when you thought you knew it all.

I have recently found myself in a search for the perfect job. What might that be? Well, I would say a job where I get to operate 200 foot tall giant robots in some cool inter-galactic robo-monkey-gladiator-swordfight type sport. I'm reasonably sure that I won't be running into one of those anytime soon. So, failing that, I think the perfect job is a Linux Systems Administration gig, working for a company that is doing something really cool, really needed, and really open. I would want to be surrounded by a group of people who are there for the same reason I am: make those systems run at peak performance, find new ways of achieving those goals, and scaling them out as needed. I need the people around me to have ideas to share, and be happy to hear mine.

Well, that's enough of the "What I want in a perfect world". The meat of this article is really to list out some of the most common questions I have been asked along this journey (which I am still on as of this writing). There are things we, as Linux Administrators, do every day. Using ls, ps, df, du, who, w, last, top, vmstat, cat, blah blah blah... but there are things that we know, we just get caught offguard having to describe them to people who ask. Hopefully this list will help you along with your search.

Warning: If you do not know a certain level of Linux, Unix, BSD, or shell scripting, this is not a cheatsheet for you. I make a lot of assumptions here, and hope that you will go off and research how to write scripts on your own.

Standard questions/exercises:

• What is an inode? An inode is a data structure holding information about filesin a file system. There is an inode for each file, and a file is uniquely identified by the file system on which it resides and its inode number on that system.

• Write a script that does "X". Using the shell you are most familiar with will help you here. We all execute oneliners every day, several times a day. Simply string those commands together in a most unelegant manner - it should not matter to the interviewer; obviously you know what you are doing. Example - a script that counts up to 10 without using count:

#!/bin/bash
ten=0
while [ $ten -le 10 ]
do
echo $ten
ten=$((ten+1))
done

• What tool would you use to detect the OS of a remote machine, and what is the command line to do it? Tricky. Could use rsh in an insecure environment and do: rsh -l username hostname "uname -r" ... or, the equivelent in ssh, assuming ssh keys are in place; ssh hostname uname -r ... or, and this would be my first try - install nmap and do a sudo nmap -O -v hostname
• Show the failed attempts to login via ssh. It's dirty, but it's a first pass that gives an indication of who tried to pop the top off your box: sudo grep Failed /var/log/messages
• What system call signals are available in linux? If you can memorize and ramble off 64 available signals, and their names.. maybe you should get out more. I rely on "man kill" and "kill -l" (thats a lowercase "L", not a ONE)
  
• How do I find what network connections are active, and what program is using them? netstat -ap -or- netstat -anp (if there is a problem with name resolution, the "n" will use the ip rather than perform a hostname lookup)
• What are setuid/setgid in relation to file permissions? Yikes. I love the short questions with the long answers. Don't stumble, mutter, or ramble here - it's easy to do. setuid on an executable file allows a user to run that program with the permissions of the owner of the file rather than the person who executed it. setguid is the same thing; only with the permission of the group the file is a member of.
• What is init? Init is the program that runs after the kernel loads that spawns all other processes. It runs as PID 1, and on RH, CentOS, Fedora, looks to the rc#.d/* files for stuff to launch. The # is the current runlevel, and the * is anything executable that is in there - either symlinked by hand or using something like chkconfig --level ##### PROGRAM on/off
• What are the standard runlevels? This is totally tricky, there's only 3 runlevels that are the same across all UNIXs/Linux's/BSD's - 0(Halt), 1(Single user), and 6(Reboot). Typical of the newish modified systems (Redhat and its derivitives), standard runlevels 3 (no X), and 5 (X) are default - set in inittab.

I need to get to a few things.. I didn't realize how long this list was going to end up being..

Trapping in bash

Scenario: You are running a code deployment to a small server farm of maybe 10-15 servers. Too small for something like tentakal, but too big to just ssh and deploy, logout, wash - rinse - repeat.

So you write a simple while; do loop and roll your stuff out that way.

The problem comes up when you realize that the deployment is hanging on a particular function, and you want to exit from that session, but want to continue your deployment on the rest of the machines - or whatever...

Enter my friend "trap".

Trap can pick up a variety of signals, and act on those in whatever way you wish.

Example: You want to stop someone from hitting ^C:

#!/bin/bash

trap "echo 'tsk tsk - no ^cing!'" 2
while true; do
echo "I dare you to press ^C"
echo ""
sleep 15
done

So - this will wait for the signal 2 (INTerrupt), and print tsk tsk - no ^cing! to the screen everytime you hit ^C.

Now, how do you stick this into a script so that it will execute that portion until it gets a ^C?

#!/bin/bash

# some code.. do what you want here...

#Trap code:

# First trap - if ctl-c is hit, it will go to the next function:
while [ trap -ne 2 ] ; do
# your commandline/script code here
done


# The next function:

while [ trap -ne 2 ] ; do
# your commandline/script code here
done

...and so on...

PCLinuxOS on the Dell Inspiron 1526

The Dell Inspiron 1526 was too inexpensive for the power to pass up.. but I did not want to run Windows on it, and I didn't want to run Ubuntu (nothing against it, just not my cup of tea). I also didn't want to force my favorite linux (Slackware) to work on it, since this was to be used for everyday work.

PCLinuxOS seemed to offer exactly what I wanted.

I installed it and it just worked. That is, except the wireless. If there was anything that I would have be an issue, it is the wireless - since there are so many ways to hack that to work.

Step-by-Step (as root, to avoid having to sudo every time):

1) echo -e 'blacklist bcm43xx\nblacklist wl' | sudo tee -a /etc/modprobe.d/blacklist

2) wget ftp://ftp.us.dell.com/network/R174291.exe

3) mkdir bcm-drivers

4) cd bcm-drivers

5) unzip ../R174291.exe

6) ndiswrapper -i bcmwl5.inf

7) depmod -a

8) modprobe ndiswrapper

9) echo 'ndiswrapper' | sudo tee -a /etc/module

10) Click Control Center -> Network -> Set up a new network interface -> Wireless

11) Follow the prompts, and use the Broadcom driver that now shows up (or should)

12) You should be taken through the process of configuring the wireless network.

13) In your terminal window, as root, type: ndiswrapper -ma

14) in your terminal window, open /etc/rc.d/rc.local and add:

depmod -a
modprobe ndiswrapper
ndiswrapper -ma

15) All done. Now when you reboot, your wireless interface will be there.

Five favorite tools

Everyone has their lists. Here's my list of "top 5 tools every linux admin needs to know intimately":

5) screen ( http://www.gnu.org/software/screen/ ) I love the ability to log in to a box, work a bit, detach my screen, go home, re-attach to the screen, and it's like I never left the session (technically, I didn't)

4) lsof ( http://people.freebsd.org/~abe/ ) - gives info about all open files. This is one of the standard troubleshooting tools for finding network issues, "why cant I rm that file?", and a host of other annoyances.

4) strace ( http://sourceforge.net/projects/strace/ ) - need to find out why a running process is hanging? strace -p [PID] that bad boy. Like lsof, I cannot live without this.

3) splitvt ( http://www.devolution.com/~slouken/projects/splitvt/ ) - I am always needing to compare two files side by side, and sometimes I just don't want to go through the series of logins to get to a specific machine to do it. SplitVT will give you multiple usable windows to work in inside of a single VT. Very cool.

2) awk ( http://www.gnu.org/manual/gawk/ ) - when im in a situation where I want to grab specific data from something, this is my best friend. Like when I need to grab the MAC address of my eth0 device: /sbin/ifconfig eth0 | grep HWadd | awk '{ print $5 }'

and finally - Rick's number one.. the single most oft used utility I have.. and the one that is going to get me branded as being on "THAT SIDE" -

1) vim ( http://www.vim.org/ ) - vi .. improved.. :) it's a tool; it's my preferred tool for editing nearly anything (except this blog, cause the web interface is also a tool - and handy) I can't expand on this any further.. either you use it or you don't..

:wq

Locking down a RedHat based box

It has been a long time since I wrote anything here - I've been extremely busy. We're doing some pretty darn cool stuff at work technology-wise, which keeps me excited (it helps when a fellas boss is a tech junkie, and like to see your ideas and then add to em..). So, yesterday I had to harden a box for an undisclosed purpose, and it made me think "Hey! I can stick this on the blog!"

This should work with any RPM based (redhat, fedora, centos, whitebox, etc) Linux distro.

Here's a not so brief overview: First, we need the local (non-root) account created, make sure we're booting to runlevel 3, drop the number of tty's to 2, lock the filesystem, remove unimportant services, fixup (not in that special IOS way, but in the southern phrase way) a few of the etc/* files, lock down some not-so-secure services, make ssh a little more secure, removing a metric good lot of un-used RPM's, locking down the services we do use, perform a little tcp hardening, some IPTables Rules.. eh.. a lot of stuff.

UPDATE: 06/02/2008: ... I wrote that about 2 weeks ago - since then, I've whipped this up: http://freshmeat.net/projects/lockdown/
It needs some outside eyes, a little TLC, and some good ol' community contribution to be worthy...