What's in my podcast roll

I’ve always been interested in learning and developing my skills, and over the past couple of years i’ve been increasingly targeting my professional development.  On a long commute, a great way to explore different areas of interest is listening to podcasts.  Here are a few of the things i’m listening to at the moment:

  • SANS Internet Storm Centre daily podcast [feed] – If there is only one podcast that should be on every IT professional’s podcast roll, this is it.  Johannes Ullrich‘s daily recap of security & Internet news is my “must listen” podcast.  Pros: Fast-paced, no fluff, edits out blank spots; very balanced coverage – i value Johannes’ opinion on almost every security topic; his enthusiasm is infectious – you can even hear his voice pitch getting higher and more excited as the podcast goes on.  Cons: none unless you like being uninformed or don’t like German accents.  Released every weekday (Johannes never seems to get sick; my son thinks he might be a cyborg 🙂 and usually around 5 minutes long.  I would pay for this podcast if i had to.
  • SANS ISC Monthly Threat Update [feed] – This is a long-format security update from Johannes Ullrich and sometimes a sponsor representative.  Pros: more in-depth coverage of certain topics, e.g. IPv6, DNSSEC.  Cons: covers many of the same news topics that have already been covered in the daily podcast; based on a webcast, so sometimes you have to go back and look at the slides to make sense of it.  Often a fair amount of time is spent on the monthly Microsoft Patch Tuesday update, which may be good or bad depending on your interests.
  • Risky Business [feeds] – This is an Australian podcast focused on security.  Pros: in-depth interviews with world-class guests who offer varied and fascinating insights; considerable time devoted to opinion and analysis of news; Patrick Gray is quite accessible on Twitter, and actually replies to messages meaningfully; quirky and sometimes outstanding full music tracks as outro.  Cons: Patrick and Adam/Mark laugh at their own jokes a bit too much and are a little obsessed with lulz; the electronic fart track for theme music makes me want to stab my eardrums out (tip: skip the first 30-40 seconds).  But the biggest con is that it’s not suitable for work – the podcast contains frequent coarse language and not all of it is beeped out.  I would love to share some of their material with my wife and kids, but references to anal rape in prisons goes beyond edginess to just plain poor judgement.  (Have you ever actually been into a prison where this was a risk, Patrick?  It’s really not a laughing matter.)  This isn’t enough to make me stop listening, but it’s a definite audience-limiter.  Make sure you subscribe to the RB2 feed as well – it covers AusCERT conferences and other similar events.
  • Cisco TAC Security Podcast [feed] – Four to six Cisco security TAC engineers talk news, tips, and products.  Pros: efficiently edited; engaging hosts, not boring; useful practical information (i’ve listened to their IPsec troubleshooting podcast more than once whilst working on a VPN problem).  Cons: feels a little too scripted sometimes; very Cisco-centric (understandably), but this means a lot of focus on ASA, which to my mind is not a compelling firewall platform.
  • Packet Pushers [feeds] – This podcast revived my interest in networking, and is a major contributor to my renewed interest in certification and professional development.  If you’re new to the podcast and like it, it’s well worth subscribing the to the all audio feed, starting from the beginning and listening to all the episodes.  Until very recently (with the Cisco Nexus series and the various SDN-related episodes) there was very little repetition of topics.  Pros: experienced presenters; world-class guests; never boring.  Cons: Greg Ferro dominates a bit too much, often talking over the top of his co-hosts; a little Cisco-centric; sometimes too many presenters, which means not everyone gets a chance to share their expertise.  Greg, Ethan, and most of the guys with a Cisco background are somewhat uninformed about Linux/Free Software/Open Source licensing, values, and culture.  They can also be uncritically fanboyish when it comes to Fruit Company laptops and tablets.
  • The Linux Action Show [feed] – A desktop/mobile/gaming-centric podcast in both video and audio formats.  Pros: lots of different news from the Linux world; CC-BY-SA licensed (o/); assumes Linux on the desktop is normal (!); metal-styled theme music is easy on the ears.  Cons: Bryan Lunduke seems to think he’s a cross between James Hetfield and Patrick Warburton, and only occasionally pulls it off; could be edited more compactly without losing its feel; a little too much material released.  I find it a little hard to keep up; a weekly podcast should be targeting around 30 minutes, in my opinion.
  • Tuxradar [feed] – a UK-based Linux news & opinion show from the creators of Linux Format magazine.  Pros: good format which incorporates listener feedback; strongly committed to Free Software from a socio-political perspective as well as a technical perspective; as expected, doesn’t assume you’re a freak if you run Linux on the desktop.  Cons: audio often muffled, slurred, and crowded with random chit-chat & laughter – this is as much a result of poor diction and too many presenters as it is of low budget or technical issues.  Released fortnightly, which is about right for a long-format show.
  • DevOps Cafe [feed] – DevOps news, interviews, and opinion.  Pros: good interviews with people from various perspectives, creative commons licensed.  Cons: a little too concerned with what’s cool & trendy in the industry (which one might argue is inevitable with DevOps).
  • The Cloudcast [feed] – Cloud industry news and opinion.  Pros: helps to de-cloud (bad pun fully intended) some of the vagueness around many *aaS (… as a Service) terms.  Cons: assumes cloud is good, doesn’t get into much technical or philosophical discussion about why we should use cloud.
  • Geeks and God [feed] – God meets nerds.  Pros: provides a unique perspective on technology that is more aware of “soft” issues (and not just spiritual ones).  Cons: content is a bit Drupal-centric (not a con for me, but if you don’t use it you might want to skip the drupal spotlight section); podcast has gone a bit quiet lately (the old team of Rob Feature and Matt Farina was prolific and particularly well-practiced at keeping discussion flowing and making a podcast interesting).
  • Andy Stanley – In my opinion, the best Bible communicator of our generation.  Even if you don’t consider yourself a Christian or “religious”, he’s worth a listen to moderate the fringe elements of Christianity who seem to dominate popular media.  His content comes in several different podcasts:
    • Your Move – a “best of” series; tends to be shorter than the weekly podcast.
    • North Point Weekly Podcast – I’ve often found the topics in this one more interesting than the “best of”.  Not all of the messages are from Andy Stanley, and i find them rather bland and traditional by comparison.
    • Leadership Podcast – good insights and the some interesting interviews on various leadership topics, not just church leadership.  Released monthly, usually around 30 minutes in length.

Some shows i’ve previously had in my podcast roll:

  • No Strings Attached – Wireless networking podcast.  I found this too slow and too scripted and got bored quickly.  I probably could have given them more of a chance to impress me, but i didn’t even make it through one episode.  No doubt as it went on it got better, but i’m less involved with wireless networking than i was previously, so it hasn’t made it back onto my radar.
  • FLOSS Weekly – Randall Schwartz’ Free Software podcast; felt too slow and scripted to me.
  • Linux Weekly News – verbatim reading of the LWN.net news articles; a bit dry.

What’s missing?  One thing that i haven’t found is a podcast that deals with system engineering in the enterprise from a Linux/networking perspective.

Source: libertysys.com.au

Fun with Linux server migrations, part 2


Seeing progress of a long-running Linux process

During the server migration mentioned in part 1, i wanted to see what a long-running rsync process was doing.  Because we had done several presyncs of the data before the outage window, there was not a lot of progress for rsync to report; it was simply churning through files checking to see which ones had changed.

The usual tool for this is strace, which shows all system calls made by a process.  You can attach it to a running process with strace -p PID, where PID is the numeric process id.  I ran strace briefly to find out what system calls rsync was making, and found that it calls lstat64 for each file.  But because it had to look through so many files, i couldn’t very well run strace -p PID 2>&1 | grep lstat641 because even that was too much data.  (I was connected to the system via my home ADSL connection, and with hundreds of thousands of files to copy, it would never have kept up with the trace output.)

So i started looking around for the right tool to sample the data without overwhelming my slow connection.  I considered writing a quick awk script, but it turns out that it’s even easier than that: sed has a built-in function for operating on the Nth line of any input file.  The general form is sed -n 'M~Np' file, which prints every Nth line starting with the Mth.  (In my case, i was reading from a pipe from strace, so there was no file.)  I tried a few different combinations and settled on strace -p PID 2>&1 | grep lstat64 | sed -n '1~10000p', which samples one in every ten thousand files that rsync processes.

I need to do this quite often on running processes, so sed -n 'M~Np' is going straight to my pool room for helpful little Linux recipes.

1. The 2>&1 is necessary because strace sends trace output to standard error rather than standard output.
Source: libertysys.com.au


Fun with Linux server migrations, part 1

Server migrations with file system structure changes

Last night i completed a P2V migration of a 2 TB Linux file server.  It was running on an old IBM x306 server with cheap SATA disks, and we were migrating it to a VMware environment with a SAS-connected disk array.  This server is going to be rebuilt in the near future, so we didn’t want to use the same amount of disk space (it was only about 60% full).  Also, it was running Linux software RAID, which is not necessary under the new environment – the disk array handles RAID.

So i needed to rebuild the file systems and copy at the file level in order to migrate the server.  Preserving the old personality but allowing for a new disk layout and a VM environment requires some care.  I wanted to maximise my options in the case of something going wrong, so i made sure the system was plugged into a managed switch which i control.  Here’s the process i followed:

  1. Create a new VM with the appropriate settings, including CPU, RAM, disk, and network.  On ESXi 5, i prefer to use LSI Logic SAS emulation for disk controllers, and Intel E1000 emulation for NICs, because:
    • both of these drivers are in the mainline Linux kernel, therefore
      • you don’t end up with unmountable root file systems or unreachable networks when you first start up the VM, and
      • you don’t have to run proprietary VMware drivers at all if you don’t want
    • they seem (anecdotally) to offer improved performance over the other emulated driver choices
  2. Do a minimal install of the OS in the new VM; use a different IP address from the source server.
  3. Set up file systems as desired.  In this case, all non-system data is in /home, so i made that a separate virtual disk and created a file system on it.
  4. From the target server, Pre-sync the data in /home.  I used the command
    rsync -avx sourceserver:/home/ /home/ --delete

    The initial sync was the largest, but i ran it again several times over a week to ensure that the final sync was as short as possible.

  5. Create an out-of-band network connection to the source server.  You might already have this.  In this case, the source server had a spare NIC which i put on our network management VLAN.  Start an ssh session on the new network connection to ensure that the old system is still reachable while you’re testing the new VM.
  6. If the system runs a Red Hat-based distribution (this system uses CentOS 5), ensure that any MAC addresses are commented out in /etc/sysconfig/network-scripts/ifcfg-eth*.  This ensures that when services are cut over, the new virtual NIC is not considered a new device, but takes on the settings of the old NIC.
  7. Create an exclude file for the system data.  I used these resources from OpenVZ and Slicehost to help me come up with an appropriate list of files to exclude.  Here’s what i ended up with:

    Some of the entries in the list above are not necessary due to the -x flag on rsync, which prevents it from crossing file system boundaries, but i wanted a fairly generic list that could be reused.  This list should be a good start for CentOS 5 systems, but may need tweaking for other distros.  The exclude file lists itself because i ran the rsync from the target and did not want to lose it when copying the root file system.

  8. Ensure that an independent backup of the source server exists.  Run it just before the outage window.
  9. When the outage window arrives, shut down all services on the source and target which are not essential for the purposes of the copy.  Here’s a list of the ones i used for my system – your list will likely be different:
    service acpid stop
    service anacron stop
    service apmd stop
    service atd stop
    service autofs stop
    service bluetooth stop
    service crond stop
    service gpm stop
    service hidd stop
    service iscsid stop
    service iscsi stop
    service isdn stop
    service netfs stop
    service nfslock stop
    service nfs stop
    service pcscd stop
    service portmap stop
    service radiusd stop
    service rawdevices stop
    service rpcgssd stop
    service rpcidmapd stop
    service sendmail stop
    service smartd stop
    service smb stop
    service syslog stop
    service xfs stop
    service ypbind stop
    service yum-updatesd stop

    Some of these might seem essential (e.g. syslog), but they’re necessary for normal running of the system, not copying its personality to a new server.  The basic idea is to minimise the amount of churn (especially logging) in the file systems being copied, while leaving networking and sshd running.

  10. From the target server, run rsync with the delete flag for any non-root system partitions/LVs on the system drive.  In my case, there was a separate /var partition.  Note that the exclude file entries need to be relative to the partition being copied, so to copy /var, you might use an exclude file like this:

    and a command like this:

    rsync -avx sourceserver:/var/ /var/ --exclude-from=/root/exclude.var --delete

    Be sure to run it with --dry-run first to make sure you’re not trashing something you don’t expect.

  11. Copy the root partition/LV in a similar fashion:
    rsync -avx sourceserver:/ / --exclude-from=/root/exclude.root --delete

    The exclude file has the contents as shown in the main exclude list above.  Again, don’t forget --dry-run to test first.

  12. Now the target VM has all the settings of the original server and is ready for the changeover.  From the managed switch, disable the frontend port(s) leading to the source server, leaving the out-of-band port active.  This prevents client traffic from going to the server.
  13. After the rsyncs are finished, reboot the target VM, watching its startup with the VMware console.  There will probably be a few services that will not be applicable under VMware (e.g. lm_sensors) – you can disable and/or remove these when convenient.  The new VM should now have all the personality of the old server, including services, IP address, and data.
  14. Once you’ve tested the target server and ensured that it is performing the source server’s job appropriately, shut down the source server from ssh session you started on the out-of-band port earlier, then shut down the out-of-band port.  This ensures that even if you’re remote from the server and it is powered up again (either by mistake, or due to mains power loss and recovery), it won’t be able to interfere with the operation of the new system.

This process went very smoothly for me last night.  So smoothly, in fact, that i was a bit worried and ran a lot of extra tests afterwards to ensure that it really was successful.  Fortunately, my fears were unfounded.  😉

Source: libertysys.com.au