SPF records and Payflow Link

EDIT – Update January 3, 2013: There was a bug in my previous SPF record below, that included “ip4:”. That should have read “ip4:”. Unfortunately the initial incorrect record allowed half the Internet to pass SPF tests, so it would be better to use the corrected version or edit this list to suit your tastes. The SPF record below has been corrected to reflect this change.


If you use Payflow Link and you’ve recently implemented an SPF record with a hard-fail (“-all”), you and your customers may periodically stop receiving transaction confirmation messages from Paypal’s servers.

Your mail logs will hopefully show an error like:

2011-10-22 17:09:12 H=mx0.phx.paypal.com [] F=<REDACTED EMAIL> rejected RCPT <REDACTED EMAIL>: SPF: is not allowed to send mail from REDACTED DOMAIN

So the question is, what IP address or block should you add to your SPF record to allow PayPal’s servers to send mail on your behalf? PayPal has a range of IP addresses of course; their transaction servers are different from the email servers (apparently).

The correct answer is here, in PayPal’s list of “IP Addresses for PayPal servers”:


Since there are so many IP ranges here, my SPF record was getting too long. So I used the include: directive and created an SPF record for a subdomain (spf-paypal.MYDOMAIN.com):

“v=spf1 ip4: ip4: ip4: ip4: ip4: ip4: ip4: -all”

And then in my main domain SPF record, I added “include: spf-paypal.MYDOMAIN.com”. This allows me to easily make edits to the PayPal portion of the SPF record and keeps it separate from my main SPF record. PayPal makes IP address changes on average perhaps twice a year, so if you use PayFlow to send transaction receipts to your customers on your behalf and you maintain SPF records for your domain, this is something you will encounter.

At the bottom of PayPal’s post, there is an option to subscribe to the answer so you can be alerted when it changes. Unfortunately they do not show a changelog; they just update the answer, so every time they change it you have to go through and compare every IP range with your SPF record, but at least you can be alerted when there is a change.

Previously, I had looked back through some transactions and (since a recent IP address update) all of them seem to be from 66.211.x.x, so my ham-fisted approach was to add:


to my SPF record, to allow –

Android sends read receipts automatically

I was surprised recently when someone responded to a read-receipt showing when I had read one of their messages. Like many others, I turned off Outlook’s option to send read receipts automatically many years ago. Since I had just changed Exchange 2010 mail providers, I assumed that I had missed this option and simply needed to re-check it on my two computers that use Outlook.

I was wrong. After checking both Outlook installations, the option never to send read receipts was still selected. Turns out my Android phone (running Gingerbread) sent the read receipt, and there’s no option to turn this off in the default mail client.

I never send read receipts because I do customer-facing email and I don’t necessarily want customers to know when I am reading emails, particularly if I want to take some time to craft a well-written response to their inquiry. They could misinterpret the delay as lack of attention to their issue, when in fact just the opposite may be true.

I ran some tests on this, and things don’t appear as bad as they seem. Android’s mail client does not appear to send the read receipt the moment you read the message – it actually sends it along with your reply, and it appears (with my limited testing) that the read receipt actually shows the “read” time to be the same as the reply time, not the actual read time. In other words, I sent myself an email at 3:01pm, read it on my Android phone at 3:02pm, and replied using my Android phone at 3:09pm. The read receipt that came back to the originating email address showed that the message was read at 3:09pm instead of 3:02pm (the latter being the first time I actually opened and read the message).

Even so, I don’t like mail clients sending anything without my approval. For some corporate Exchange users, you might be able to have your Exchange administrator block read receipts, but it is probably a server-wide setting, so don’t be surprised if you encounter some organizational push-back, or at least a lengthy discussion about it first.

In my case, I happen to route all my mail through a primary server that runs Exim first, before passing it along to my main Exchange server and a backup server. So I added the following line to my /etc/exim.conf:

headers_remove = Disposition-Notification-To : Return-Receipt-To

I added that line to my “virtual_aliases_nostar:” director, right below the first line (“driver = redirect”).

Then don’t forget to restart Exim.

I know this isn’t the very best place for it, because it doesn’t work 100% yet, but it’s a start. What happens is that senders still get delivery reports, but read receipts are no longer sent by my Android phone even when I reply. Looking at the message headers, the Disposition-Notification-To: header and Return-Receipt-To: header are no longer present in messages I receive when the line above is present in my exim.conf file. Ideally the line would be placed in a location that would immediately strip those header lines and prevent both delivery reports and read receipts, but as I mentioned I don’t mind someone knowing that their message made it to my server OK.

Since my server is running cPanel, I’ve done the ultimate evil by editing exim.conf directly in the shell; I will need to figure out how to get this line into the cPanel configuration editor and I’ll post back when I do so.


in cPanel, you can copy /etc/cpanel_exim_system_filter to a new file (I called mine /etc/MYDOMAIN_exim_system_filter where MYDOMAIN is my actual main domain name), then add the following to that file:

headers remove "Disposition-Notification-To"
headers remove "Return-Receipt-To"

This will remove the headers and not be overwritten with cPanel updates (if you leave the default cPanel filters file in place it does overwrite, so be sure to work on a copy).

FVS338 dialup backup fails

If your Netgear FVS338 dialup backup is failing, you might have a firmware problem. Sometime after firmware 3.0.2-21 (and not fixed as of 3.0.7-24), the dialup interface connects and grabs an IP address – it even allows ping, traceroute, and DNS lookups from the FVS338 web interface – but it will not route any traffic from the LAN to the WAN. Your client machines will be able to perform lookups, but no other traffic will route, so the dialup backup is useless.

The solution is to downgrade to an earlier firmware, which you can do from here. Click “Downloads”, then click “Show All Versions” and select an earlier firmware. I can confirm that dialup backup mode works as of 3.0.2-21; if you try a later one and it works, let me know via comment.

Apache: “couldn’t create child process: 720005”

If you installed Apache on Windows (I like WAMPServer myself) and you’re trying to run Perl scripts, you might encounter the dreaded Internal Server Error (500) and a cryptic entry in Apache’s error log:

[Sat Jun 04 19:01:50 2011] [error] [client] (OS 2)The system cannot find the file specified. : couldn't create child process: 720002:
[Sat Jun 04 19:01:50 2011] [error] [client] (OS 2)The system cannot find the file specified. : couldn't spawn child process:

I realized I had installed ActivePerl to the wrong place, and Apache couldn’t find it (I was able to run my Perl scripts from the command line, just not from Apache). All my perl scripts start with #!/usr/local/bin/perl as the shebang line, since this is what my Linux servers all use. So on Windows, I always install ActivePerl to c:\usr\local and it works great. My earlier mistake was to install ActivePerl to c:\usr\local\perl. So by removing and reinstalling ActivePerl to the correct directory, I thought this would be solved.

Then I was getting:

[Sun Jun 05 07:29:24 2011] [error] [client] (OS 5)Access is denied. : couldn't create child process: 720005:
[Sun Jun 05 07:29:24 2011] [error] [client] (OS 5)Access is denied. : couldn't spawn child process:

Now I was scratching my head. Access denied? Turns out, even though I had uninstalled and reinstalled ActivePerl to c:\usr\local, there was still a “perl” folder inside c:\usr\local\bin from the earlier installation of ActivePerl. Of course my scripts are looking for \usr\local\bin\perl – so they are finding the FOLDER called “perl” rather than the executable, perl.exe. You can’t execute a folder, hence the access denied error. Once I deleted the erroneous folder, the shebang lines in my scripts directed Apache to the perl executable and my script started working immediately.

So, if you are seeing this error, look very closely at your shebang lines and where they point to. In my case, uninstalling ActivePerl did not fully remove the prior directory structure and that caused problems even when I later reinstalled it to the correct place.

Munin and High Loads

Munin, while a great monitoring tool, can be a little bit of a CPU hog. I had installed it awhile back and looked at the graphs just for fun now and then (I’m lucky, my servers are humming along pretty smoothly), but I started noticing load averages were measurably higher than they were before I was using Munin.

You can’t just stop the munin-node service in some cases; if munin-node graphs are generated by a cron job, you have to stop the cron job too. I had an older CentOS server that required removal of the cron job for the munin user, but on a newer Ubuntu server all I had to do was stop the munin-node service the munin cron jobs had squirreled themselves away inside /etc/cron.d/munin and /etc/cron.d/munin-node.

See if you can guess what time the munin-node was stopped:

I decided to just run without Munin for awhile; I’ll turn it back on someday if I need it to diagnose something specific, but I don’t see a compelling reason to run it 24×7 just chewing CPU cycles.

Motorola Xoom USB Dock

I took the stock Motorola Xoom dock (the cheaper one, not the speaker dock) and modded it to provide a USB link to the PC. The mod does not destroy the dock, but it does render the audio-out jack unusable until the mod is reverted. I don’t care about audio-out on my dock, but I do care about being able to drop the Xoom in the dock and have it connected to my PC so I can transfer files easily. Here are the steps and photos. Continue reading Motorola Xoom USB Dock

Android Voice Dialing with Confirmation

If you have an Android phone, you may be lamenting the fact that when you use bluetooth to launch a voice prompt and say “Call mom at home”, the phone mysteriously starts dialing Kentucky Fried Chicken in Sandusky, Ohio (and your mom happens to live in Florida). And it does all this without confirmation! Thanks, Google.

To the rescue: Cyberon Voice Commander for Android. I found it in the market; it gets the contact name right probably 90% of the time (unless I’m in a very noisy environment), and of course the biggest feature of all, it confirms that you want to call what it thinks it heard you say. Continue reading Android Voice Dialing with Confirmation

Google Hacking Me

Update: they haven’t done this in awhile now – maybe they stopped this behavior (October 2011).

(original post follows)

I know this is probably under the radar for most people, but Google, who says “do no evil” tries to hack web sites on a regular basis. Last night, Googlebot (more specifically the host “crawl-66-249-68-246.googlebot.com”) tried to send a made-up QUERY_STRING variable to one of my web site programs. (For those who don’t know, the QUERY_STRING is the stuff starting with a “?” in a long url; it is a way of passing information to a program, similar to submitting a web form). Googlebot tried to send a completely made-up “action” parameter in an effort to see how my script would respond.

Of course since I know what I’m doing, my script kicked Googlebot off and did not reveal anything, but it still ticks me off that a company who says “do no evil” goes poking around in what can only be a nefarious manner, since there is no legitimate reason to pass random parameters to web scripts.

Countering a Slow Loris Apache attack

Here is a poor man’s way to mitigate a slow loris attack. The DoS attack, not the fuzzy mammal with the poisonous elbows.

If you’re reading this, hopefully you’ve searched and figured out what the slow loris attack is all about. You probably also run Apache 1.x or 2.x. I read about the antiloris module (and noloris as well, IIRC), but I wondered if I had tools already in place to help mitigate another attack.

First, you need to lower the “Timeout” value in Apache. The default is 300 seconds, which means you are holding a socket open for 5 full minutes. Since what day in the mid-1990s has that really been necessary. And if you have scripts that run that long, you have another wide open avenue of DoS attack that will be exploited soon anyway.

Second, configure CSF to use the “PORTFLOOD” feature (requires ipt_recent), and set a reasonable limit on tcp connections to your web server port(s).

When I ran slow loris against the server in question, clearly just reducing the timeout value had a major impact on slow loris’ ability to keep the server down. Response time slowed considerably, but Apache was still serving pages. Adding the CSF PORTFLOOD rule stopped slow loris dead in its tracks.