Hideki A. Ikeda (HidekiAI) [池田英紀] ["Tony" Ikeda] – BLog

Most people pipe and combine stderr to stdout and send it to either /dev/null or other log file, but there are times when you want it separated.  There are times when you only want to preserve just the errors and not care about debugging or info type of messages…

Following are examples of  different combinations of redirections.  The example tries to “ls x” in which a file (or dir) “x” do not exist in my home (“~”) directory…

hidekiai@devbox ~ $ ls x  2>/tmp/std.err 1>/tmp/std.out ; ls 2>/tmp/std.2.err 1>/tmp/std.2.out ; ls x > /tmp/std.both 2>&1 ; tail /tmp/std* 

==> /tmp/std.err <==
ls: cannot access x: No such file or directory

==> /tmp/std.out <==

==> /tmp/std.2.err <==

==> /tmp/std.2.out <==
Desktop
Documents
Downloads
Music
Pictures
Public
Templates
Videos

==> /tmp/std.both <==
ls: cannot access x: No such file or directory

First and second examples directs output to two files.  First example is with error and second example is with no error. The 3rd example combines both stderr and stdout and pipes it to “std.both” file…

OK, just for clarity for those who do not want to strain their eyes on the “;” separated statements, here they are separated…

hidekiai@devbox ~ $ ls x  2>/tmp/std.err 1>/tmp/std.out

==> /tmp/std.err <==
ls: cannot access x: No such file or directory

==> /tmp/std.out <==

hidekiai@devbox ~ $ ls 2>/tmp/std.2.err 1>/tmp/std.2.out
==> /tmp/std.2.err <==

==> /tmp/std.2.out <==
Desktop
Documents
Downloads
Music
Pictures
Public
Templates
Videos

hidekiai@devbox ~ $ ls x > /tmp/std.both 2>&1
==> /tmp/std.both <==
ls: cannot access x: No such file or directory

I have several servers that are Gentoo based because these machines are either fast enough to build as well as homogeneous nature of the boxes to allow me to build it on build server, and use NFS to mount /usr/portage/packages on other hosts, and use “–usepkg=y” to just use the same binary package built by the build-server.  As long as the package.mask and package.use match all across, I can use the binary packages built on build-server and never have to build it anywhere else.

In any case, because of the etiquette to not wanting all my boxes to “emerge –sync” from WAN servers (gentoo site even says some mirror sites will ban your I.P.), I have the build server “–sync” to the WAN, and set the internal machines to rsync from the build-server.  This used to work great for many many months (more like years) that I’ve been rysnc’ing from local server, but just recently I’m not sure what happened but it all broke…

What’s interesting is that these clients can “emerge –sync” (or eix-sync) just fine from WAN address.  I can also manually call rsync from the client end-point to the LAN rsync (build) server and I can manually rsync files that are usually rsync’ed via emerge just fine…  Before I go on any further, here’s what I’m seeing…

On the build server side:

2011/12/04 10:17:05 [27897] rsync: read error: Connection reset by peer (104)

2011/12/04 10:17:05 [27897] rsync error: error in rsync protocol data stream (code 12) at io.c(764) [Receiver=3.0.9]


On the client side (3 seconds later):

2011/12/04 10:17:08 [13506] rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(549) [Receiver=3.0.9]


As you’ve noticed, both server and client are using rsync v3.0.9 so they match in protocol (I hope).  If you do “emerge –sync –debug –verbose” you’ll also notice that “–timeout=180″ (that’s 3 minutes) is set for rsync.  I’ve also manually set the “–contimeout” in the PORTAGE_RSYNC_EXTRA_OPTS (on the client side) just in case.

What really bugs me are the fact that:

1. When the clients use WAN “rsync://” address, “emerge –sync” works just fine
2. Manually calling “rsyncrsync://mylocalserver/gentoo-portage” to sync files from my build-server works just fine

After some more fooling around, I’ve managed to reduce this problems on some of my servers by routing to the static I.P. address that is of the fastest connection on my LAN as well as not using DNS (i.e. “rsync://hostname_of_server” versus “rsync://x.y.z.w”).  This basically gave me the confidence that it wasn’t rsync (clients), rsyncd (server), or emerge options that was causing the problem, but the problem is not solved since even with DNS lookup eliminated, it still had timeout problems…

So what is bothering me again has shifted, this time towards wondering why what worked few days ago suddenly stopped.  Is it my DNS?  My routing metrics?  As mentioned, it fixed most of the machines, it didn’t fix all.  One of my dev-box who’s not on the server routing VLAN has to still use the slower route, and it times out on the average of 15 – 20 seconds.

I must first admit that I’ve been playing around with Fedora’s 389 Directory Service for a while now.  On that box, I’ve also setup dhcpd and named as well.  But the truth is, most of my name resolutions on the VLAN is through each machine’s /etc/hosts file (this is why I’m trying to get an LDAP/DNS/DHCPD/KDC combinations going – similar to what I’ve been spoiled with on Active Directory), but until then, I’ve been either hand-editing or copying files to do name resolutions.

I’m pretty sure in the end, it’s something in my network slowing things down, with so many different points of failures it’s hard to track down.  Reducing point-of-failures is an art in server development, which I won’t go into details here, but if you’ve done a lot of server coding, you know what I mean.

In any case, I was about to build rsync’s source with some printf()’s or something to track the issue since I have half-convinced myself that the emerge Python script was timing out…  But my laziness of doing this lead me to try something else and I’m glad I did…

I decided rather than debugging rsync, I’d first replace rsync with bash script as a proxy like so…

$ cd /usr/bin
$ mv rsync rsync.bin
$ nano rsync

… and on the file:

#! /bin/bash
# replaces /usr/bin/rsync -> rsync.bin
# We'll show what got passed for debugging...
echo rsync $@
# We'll call what was requested but with strace to see what's going on
/usr/bin/strace /usr/bin/rsync.bin $@

… of coruse, want it executable

$chmod +x rsync

… and ran it

$emerge --sync

I didn’t discover much…  All I discovered was the fact that after emerge triggers rsync and the Python script times out, I get a response back from rsync server.  I’ve also noticed that in some cases, it does the 2nd pass at calling rsync.  It turns out that first rsync call is to grab the timestamp.chk file update and second call (if timestamp is old) is to actually grab the portage tree…  If the first rsync takes longer than 15-16 seconds, I’d not see the second rsync request and the “timed out” message comes up…

That basically told me that it’s not rsync that’s the problem.  My original suspicions were that perhaps rsync forked a thread and while the thread blocked receiving data from the rsync server, the main thread timed out.  But no, rsync is running fine, it’s the Portage Python script(s) that seems to just impatiently walk away rather than block and wait for rsync to return…

With that in mind, I decided to look at the file actions.py (located at “/usr/lib64/portage/pym/_emerge/” path) and searched for anything that’s calling “rsync”.  I noticed that most of my timeouts are in the range of 15-16 seconds, and there’s a environment variable PORTAGE_RSYNC_INITIAL_TIMEOUT which was defaulting to 15 seconds…  Jackpot!  I set it to 180 seconds (I  could probably lower it to like 30 seconds) in my /etc/make.conf file like so:

PORTAGE_RSYNC_INITIAL_TIMEOUT="180"

Well, that solved it… Now, after knowing what I was looking for, if I searched on the net for “PORTAGE_RSYNC_INITIAL_TIMEOUT”, I found two links that you’d probably want to read:

• http://www.linuxhowtos.org/manpages/5/make.conf.htm
• http://forums.gentoo.org/viewtopic-t-622757-start-0-postdays-0-postorder-asc-highlight-emerge+sync+sigterm.html

Mystery solved…

Note: If you were to consider something like this as a “bug”, the bug is that the Python script (actions.py) doesn’t block when it calls rsync, nor does it poll the status returned from rsync.  What I’ve observed is that the Python script bailed out first, then rsync finally unblocked (and almost always in success) but it was too late…  I don’t think it’s a design flaw though…  How many times have you debated on a design to whether you should block, poll, or just timeout on an RPC method?  I think my situation is just an edge-case since Portage has been functionally working for quite some times…

First and foremost, this is just a disclaimer of few things:

• I’m sure there’s some agreement in the EULA that says you can’t make a fake account and have to be honest about your user information between you and Microsoft/Skype…
• This isn’t a “real” security camera, if you have half the brain to setup Skype (such as setting auto-answer, auto-launch, etc) you’d realized that.  It’s not like having this is going to make your house secure.
• This is just a hack!  I can’t make it work on my Linux box because I don’t have gdm/xdm/lightdm or any kind of DM in my rc.local (or /etc/init.d) scripting to auto-launch it.  Yes, I can ssh to home from remote, port forward to VNC to my box, turn on Skype, etc, but I’m lazy.  So this is with Windows (and so it may not be accurate instruction since I don’t run one at home).

So it’s a hack, but it is a very quick-and-dirty way to get something going to let you monitor (video and audio) of your home over Skype (i.e. Android phone, work station at work, etc).  I don’t use Skype much, but IMHO it’s a very solid technology, and it’s UDP based, to top it off, secure.  No wonder Microsoft bought the technology.  I hope it keeps maturing, and I hope Skype never dies.  Although I’m really hoping they won’t cut off Linux…

Firstly in order for this to work, you probably want to make sure you have your Windows auto-login in case you have a brown-out.  I like tweak-ui, I’m not sure that works on Windows Vista or 7…  But I’m sure you’re smart enough to figure out a way so that if you have a brownout and your machine reboots, it’ll be back in the desktop…  I’m not sure how Skype reacts to screensavers though…  In Linux, I can ssh to my box and kill the xscreenserver…

You’d need another Skype account, this account you’d not want to give it to anybody.  Just between your phone (or desktop or anywhere remote that you’d use your Skype) and the box which you have Skype video.  You then want to set the Skype option to auto-start the video upon answer…  This is why you only want yourself on this account (probably should set only answer and start video on my list)

Set Skype to auto-start when Windows starts.

Now point that webcam to where you’d want to be monitoring when you are remote…

There are few pointers on why I think it’s goofy but useful:

In any case, there you have it…  A poor-man’s security camera *grin*