This post is authored by our Senior Operations Engineer, Erik Hollensbe.
/proc filesystem is one of the easiest, cheapest ways to quickly get at
the status of your machine. We use it extensively in
preferring it to other methods of retrieving data.
Something to get your feet wet
netstat -an is a great way to get a list of all the connections running on a
machine, but as mentioned in the previous article, it can be quite slow as it
pulls everything up. How can we quickly get the count of all network
connections on the machine, for example?
/proc/self/net/ has several files named
tcp6, etc that
contain this information. They’re actually in each pid-named directory, but
contain the information for all connections it can see.
/proc/self is simply
a magical directory that points at the current process id, similar to how
/proc/$$ would work. This allows us to interrogate these files without being
Here’s some code to get that count:
1 2 3 4 5 6 7 8 9 10
Which on my dev machine returns
255. The output from
netstat -an (with
descriptive headers) reports
261. We’re good.
Read the friendly manual
man 5 proc has tons of information on these files. I strongly suggest you
at least page through it before trying to write C to interrogate the system
directly. Depending on what you need to do, reading these files could be
considerably less error-prone code.
Let’s dig deeper.
netstat -anp yields all sockets and the PIDs and executables attached to
them. This is very useful when trying to find a problem
bind(2) or a process
which is not cleaning up its sockets properly.
Output looks something like this:
1 2 3 4 5 6 7 8 9 10
Let’s write this, using /proc.
exe and cmdline
/proc/<pid>/exe are two very useful ways to get at
process information for any given process ID.
cmdline is just a text representation, with each argument delimited by a NULL
(ASCII 0) byte, accessible by all users. This is the “process title” you see in
exe is a little more esoteric but extremely useful. Only accessible by the
user (or root), it’s a symlink to the dereferenced path of the binary used to
run the process.
/bin/zsh -l as my shell.
cat /proc/self/cmdline at this point is
/bin/zsh\x00-l (the escape is not literal — that’s an ASCII 0 byte). However, on
Ubuntu machines zsh comes from
/bin/zsh4, which if you
/proc/self/exe, is what you end up getting.
exe is really nice for things like
sendmail which change their
process title; the contents of which will be reflected in
instead of the actual binary executed.
OK… Get on with it.
pgrep first, shall we? You’ll need to run this as
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
Quite a bit slower than
pgrep, but very effective. Can you write a version
/proc/*/cmdline and works exactly like
Sockets and readlink
readlink is a swiss army chainsaw. It’s great for locating the source of
symlinks, device files, and … sockets.
Let’s look at a running copy of
named. It has numerous file descriptors in
/proc/1191/fd and we want to find out which ones are sockets. How do?
Well, readlink can tell us (again, you’ll need to be the same user as
root to try this):
1 2 3 4 5 6 7 8 9 10 11
Which should yield output similar to this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
So, we see
/dev/null in quite a few spots (this is a daemon, after all), and
some special syntax for ephemeral file descriptors:
These inodes can be used to look up values in the network files we examined earlier.
Finally! We’re getting to the meat!
If we look at
/proc/self/net/tcp, we see this:
1 2 3 4
This contains the information we need — note that
inode column on the far
right. We can also use
/usr/bin/stat or an equivalent to cheat a little on
what we look at.
So, let’s dig these sockets up! I’ve written this in ruby instead of shell because it should be a little faster, and a little more terse. I’ve done my best to comment the code on a line-by-line basis so you can understand what’s happening.
Note that no attempt has been made to handle unix sockets or IPv6. An exercise for the reader, perhaps.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47
This is the kind of output it yields when run as root:
1 2 3 4 5 6 7
man 5 proc for more information on this great service!