• Speed Up CrashPlan Backups: Automagic!

    UPDATE

    Well…  I can no longer recommend CrashPlan, even using my fix below.   I recently upgraded my file server, performed a computer adoption according to CrashPlan’s instructions…. and CrashPlan lost all of my backups from that machine.  All 16TB or so or them.  It also lost my backup set definitions after the adoption.

    CrashPlan support was… not particularly helpful.  They claim they see the 16TB of data attached to the right computer GUID, but I can’t see the data in my client, and when I look at my account in the CrashPlan web interface, I see nothing.  Support said it “might work” if I forced a backup to run… and it didn’t.

    My CrashPlan account expires soon, and I won’t be renewing.

     

    Another Update:

    If my fix doesn’t work for you, try changing the server that you’re connecting to, as described HERE.  Thanks to reader Jorgen for his comment with this tip!  This will, unfortunately, mean starting your backups over from scratch.  :(

     

    Reading some of the recent comments on one of my previous posts, I’ve noticed that some people seem to be having problems following the instructions there — looking at the wrong XML files, forgetting to restart the CrashPlan service, etc.   In the interest of making this fix more “user friendly”, I’ve developed some scripts for Windows and Linux that does some “automagic” — one-click fixes, basically.

    Windows:

    Download:  CrashPlanFix-Windows.zip

    Unzip all files to a directory somewhere, then run CrashPlanFix.bat as Administrator (right-click, Run as Administrator).   You’ll see a window pop up showing some progress messages.  As long as you don’t see any errors, everything should have worked properly.

    Linux

    Download:  CrashPlanFix-Linux.sh.gz

    Decompress the file and then run CrashPlanFix-Linux.sh.

    Mac

    Nothing yet — I don’t have access to any Mac systems.  :(

    (Update: 13 Feb 2014)  Thanks to reader Bill W, Mac users can now feel the automagic love.  Download his script for Mac HERE.  Note that this will need to be run with elevated privileges via the ‘sudo’ command.  Thanks Bill!


  • Followup: Speeding up Crashplan Backups

    UPDATE

    Well…  I can no longer recommend CrashPlan, even using my fix below.   I recently upgraded my file server, performed a computer adoption according to CrashPlan’s instructions…. and CrashPlan lost all of my backups from that machine.  All 16TB or so or them.  It also lost my backup set definitions after the adoption.

    CrashPlan support was… not particularly helpful.  They claim they see the 16TB of data attached to the right computer GUID, but I can’t see the data in my client, and when I look at my account in the CrashPlan web interface, I see nothing.  Support said it “might work” if I forced a backup to run… and it didn’t.

    My CrashPlan account expires soon, and I won’t be renewing.

     

     

    So, I now have a bunch more performance data from CrashPlan covering the period after I put my fix in place (from my last post).  This is just a quick followup to show that this fix does, indeed, have the desired effect.

    They say that a picture is worth a thousand words, so voila:

    crashplan-afterfix

    It’s left as an exercise for the reader to determine the point in time at which I put the fix in place.  😉

     


  • Speeding up CrashPlan Backups

    UPDATE #2

    Well…  I can no longer recommend CrashPlan, even using my fix below.   I recently upgraded my file server, performed a computer adoption according to CrashPlan’s instructions…. and CrashPlan lost all of my backups from that machine.  All 16TB or so or them.  It also lost my backup set definitions after the adoption.

    CrashPlan support was… not particularly helpful.  They claim they see the 16TB of data attached to the right computer GUID, but I can’t see the data in my client, and when I look at my account in the CrashPlan web interface, I see nothing.  Support said it “might work” if I forced a backup to run… and it didn’t.

    My CrashPlan account expires soon, and I won’t be renewing.

    UPDATE

    See my new post here for some scripts that will edit the config files and restart the CrashPlan service for you automatically!!!

    Another Update:

    If my fix doesn’t work for you, try changing the server that you’re connecting to, as described HERE.  Thanks to reader Jorgen for his comment with this tip!  This will, unfortunately, mean starting your backups over from scratch.  :(

    TL;DR:   CrashPlan’s dedup algorithm is a bottleneck on faster network connections and/or slower CPUs.  Changing a single setting inside an XML file pretty much disables the algorithm and makes things not slow to a crawl over time.

    So, I recently got a shiny new DSL line installed that has an awesome 10Mbps upload speed — 10x faster than my cable connection, and the fastest upload speed of any residential service available to me that doesn’t have ridiculously low usage caps (thanks TekSavvy for your unlimited plans!).

    I decided to take advantage of this increased network capacity to expand the amount of stuff that I backup using CrashPlan+ — due to my relatively slow upload speeds previously, I only backed up critical documents, leaving stuff like pictures from my SLR camera to my local backup system.   With 10Mbps up… no longer!   <insert BACKUP ALL THE THINGS meme image here>

    Anywho… things went great.  At first.   I added a ton of files to my backup set, and off CrashPlan went, uploading at 9.6Mbps… then 9.4Mbps…. then 9Mbps… then 8Mbps… then 7Mbps…. and down and down and down… when it hit the 3Mbps mark and kept going down (despite me tweaking the compression/deduplication/network buffer/other settings), I figured an email to CrashPlan support was in order.    The reply I received, while professional and clearly NOT just a template response, was basically “Well, 3Mbps is better than most of our users get;  it’s a shared network;  you’ve already done everything I can recommend.”   That was clearly not what I was hoping for.  I should also note that one of the things I noticed and mentioned to support was that the CrashPlan process was taking up 100% CPU time on one core, indicating it might be CPU-limited.

    So I set about gathering real stats, and thanks to CrashPlan’s built in logging… I had a whole bunch of data points.   Using a bit of grep, awk and sed (the 3 sweetest words I know next to Perl 😉 ), and a bit of Excel charting, I came up with this:

    crashplan-speed

    Hmm.  Interesting.  Anyone who’s studied computer science should be screaming “O(ln(n))” right now — this type of logarithmic decay screams “algorithm performance issue”.  You simply do not, ever, see this pattern due to overloaded network capacity.   This type of pattern, coupled with 100% CPU use, told me there was something wrong with one of CrashPlan’s features, and I strongly suspected the de-duplication functions, because I already had compression turned off (I knew most of my data wasn’t compressible) and encryption algorithms don’t decay like that.

    So… I emailed CrashPlan support again with this evidence (along with a whole bunch of CompSci geek reasoning), and was told, basically, “you’re already faster than most of our customers, if you don’t like it, find another provider”.   Yikes.

    Talking with a few other people I know who use CrashPlan with large (multi-terabyte) data sets… this seems to be a common problem.

    So… being a geek, I figured I’d check to see if I could do anything other than spend a bunch of money to upgrade the CPU in that system.

    I navigated to the CrashPlan configuration directory, /usr/local/crashplan/conf/ (this is on Linux; Windows users, you’ll have to figure out there this is yourself, sorry!), and started digging.  I stumbled across this gem in the file my.service.xml :

    <dataDeDupAutoMaxFileSize>1073741824</dataDeDupAutoMaxFileSize>
    <dataDeDupAutoMaxFileSizeForWan>0</dataDeDupAutoMaxFileSizeForWan>

    Hmm.  Interesting.  “0” is often used as a metavalue that means “unlimited”… so maybe CrashPlan will ALWAYS dedupe EVERYTHING when going over a WAN link, but only dedupes files smaller than 1GB when going over a LAN link, presumably because they recognize that there’s a balance between CPU capacity and network capacity.  It seems that they assume everyone has ridiculously slow upload speeds that are typical of most residential Internet connections.

    Being the smarty that I am, I figured I’d set it to not dedupe any files larger than 1 byte when going over the WAN:

    <dataDeDupAutoMaxFileSizeForWan>1</dataDeDupAutoMaxFileSizeForWan>

    (Note:  If you use backup sets, you will have more than one of these lines, one per backup set.  I suggest changing them all;  I have not done any testing to see what happens if you disable it only for some of the backup sets.)

    I then restarted the CrashPlan engine (/etc/init.d/crashplan restart).

    And… VOILA.    My CPU usage dropped from 100% of 1 core down to ~10% of 1 core, and my upload jumped from 2.5Mbps (and dropping) to 7.5Mbps (and holding/fluctuating between 6.9Mbps and 7.5Mbps).   I’m now seeing patterns that more accurately cover the “variable network bandwidth due to shared service” theory, without seeing logarithmic decay in performance.

    I have confirmed with other people that have noticed slowdowns on large data sets that this fix works for them as well, so I can confirm it’s not just something weird on my system.

    I updated my 2nd ticket with CrashPlan support with this information, suggesting that they expose the max-file-size-for-dedupe inside the client, rather than making people go dig through XML files.

     

    Update: Once I reached non-peak hours, I’m starting to see upload speeds >9Mbps again as well.  Yay!

    Update 2:  Check out my followup blog post with an updated traffic graph HERE 


  • Installing XenServer 6.2.0 from a USB Stick

    Most servers nowadays don’t ship with optical drives by default.  This means that operating systems need to be installed over the network or via USB.  I recently had to upgrade a XenServer 6.1.0 host to version 6.2.0.   Here are the steps I followed to get a working USB stick:

    1. Download the XenServer 6.2.0 ISO image (http://downloadns.citrix.com.edgesuite.net/7281/XenServer-6.2.0-install-cd.iso)
    2. Format a USB stick using FAT32.
    3. Use unetbootin (http://unetbootin.sourceforge.net/) to install the ISO to the USB stick.  Use the “DiskImage” option instead of the “Distribution” one, and point it right to the ISO file.
    4. Now we need to fix up some boot stuff:
      1. All paths are relative to the root of the USB drive.
      2. mv boot/isolinux/isolinux.cfg boot/isolinux/syslinux.cfg
      3. mv boot/isolinux boot/syslinux
      4. mv syslinux.cfg syslinux.cfg.bak
    5. Unmount the USB drive and eject it.
    6. Boot from the USB stick.

    If you get a “kernel not found” error at boot, double check that you renamed all the files properly.

    Now you should have a working, bootable USB stick to install/upgrade XenServer!


  • Oreo’s New Playmate

    After 7 months with our bunny rabbit Oreo, we’ve decided that he needs a bunny friend for more social interaction.  Although we give him a lot of human attention, there are stretches of the day (e.g. when we’re are work) where he’s all alone.  Since rabbits are social creatures, this isn’t great, and we think it’s actually been affecting his behaviour a bit.

    So… enter bun #2.  We’ve been kicking the idea of getting a second bun around for at least a month now, and finally decided to do it.  We’d been watching the various pet shelters (mostly Ottawa Humane Society and New Moon Rabbit Rescue) to see if a bun came in that we liked.  There were definitely a few candidates, but we decided to go see Millie at the OHS yesterday.  She was surprisingly friendly with us right off the bat (even climbing up in our laps to check us out), and even let me pick her up without any fuss.  We talked to the adoption staff to get a better sense of her temperament, habits, etc, and decided that she would be a good fit, and filled out the initial paperwork and put down a deposit.

    We set up a new dedicated pen area for her last night (OHS policy requires a 14-day quarantine / adjustment period from existing pets in the household — although in the case of buns it’s a good idea anyways since the bonding process takes time and you want to keep them separate at first), picked up all the new “stuff” (litter pan, water bottle, toys, hay hopper, etc) this morning, and picked her up this afternoon!

    She was a little hesitant about getting into the pet carrier to leave OHS (given her history, kind of understandable), but once home she didn’t take too long to get out and explore her new space!  Here are some quick pics taken about an hour and a half after we got her home.  As you can see, she’s pretty comfortable in her new space already!

    20130427_154322

    20130427_154344

    20130427_161415

    20130427_155219

    20130427_161609

    20130427_161621

    So far, she’s proving to be quite the little explorer!


  • Oreo’s Epic Habitat

    So, something that I didn’t mention during my last post is that I came back from my hunting trip with another animal other than the moose.  The special thing about this one is that it’s still alive!

    During one of our trips back into town for supplies, I went out shopping with my mom to get some dog food (among other things).  This meant going into various pet stores until we found someone that had stock on the stuff we were looking for.  One of these pet stores had an absolutely adorable rabbit for sale.   I snapped a cell phone picture of it and sent it to my girlfriend asking what she thought.   Long story short, I went back to the store later that day and bought it.    My mom kindly cared for it for the week that I headed back out for the rest of my hunting trip, then the rabbit came back home with me at the end of hunting (he didn’t much care for the 8-hour car ride though).

    The rabbit included a cage when I bought him from the store, but it’s relatively small, and we wanted him to have a bit more room.   My girlfriend dug up some info about “NIC Habitats” (NIC stands for Neat Idea Cubes, which was the original brand name for the type of grating that we used) and we decided to build one for Oreo (the name of the bunny, after his white and chocolate brown coloring).

    Several shopping trips, many hours of planning and assembling, some cursing and several hundred zipties later, we came up with this:

    It’s 3 levels of bunny fun, with a ramp between ground level and the first platform.  We’re hoping that he’ll be able to learn to jump up to the second platform from the first, but if not, we have a plan to build some “steps” for him.   The whole thing is carpeted too, for better traction for him.   The enclosure is 42 inches on each side, giving him 12.25 square feet of space on the ground level, with the platforms giving him another 10.9 square feet!

    The door swings open for both easy hopping in/out for Oreo, plus easy access for cleaning for us.  We’re going to come up with a locking mechanism later — for now we just hold the door shut with a ziptie when we want to lock him in.

    Somewhat surprisingly, he seemed to settle right into his new bunny condo after a few minutes of snooping around.  He started doing his mini bunny flops and stretching right out almost right away.  He’s a happy little guy right now!


  • Another Year, Another Year of Hunting

    My second post on this blog is also non-network-related.

    My annual vacation consists of taking 2 weeks off to go into the middle of nowhere to hunt and spend time with my dad.   And by “hunt”, I mostly mean “sit in a hammock nowhere close to a computer and read books”, at least for me, but hey, that’s just semantics.

    However, while I have little interest in waking up at 5:30AM during my vacation, my dad is a very avid hunter, and yesterday it paid off.  He came back to camp in the morning and said “I hit a moose.”    My immediate reply was “With what?” because I’m just funny like that.

    Usually when a moose is hit with something like, say, an arrow, it runs away and eventually dies due to blood loss and what not, assuming you hit something vital.  This time was no exception, and my dad tells me that the moose ran off into some dense brush, and he heard it fall maybe a few hundred meters in, but he wasn’t sure exactly where.   I’m really, really good at tracking stuff, so I tell him I’ll come help find it.  We pack up and head out.

    Long story short, we (OK, I) find the moose in fairly short order, and I’m utterly amazed:

    Yes, that arrow is in its ass. There’s barely any blood evident (we found NONE while tracking it), yet here it is, dead (I approached cautiously in case it was, say, merely resting and EXTREMELY ANGRY).

    Here’s a member of our hunting party expressing his surprise that a butt shot killed this thing:

    Turns out that it’s a pretty nice little bull, probably 3-4 years old.  He’s got a beautifully colored rack on him!

    And finally, a picture of me and my dad:

    This little bull dropped in some pretty nasty bush, and it took us quite a while to get it out.   From the time we went in to get it until the time that we had it hanging in the garage was somewhere over 11 hours (including time for field dressing, quartering, transport, etc).   LONG day, let me tell you!

    Since we switched to bow hunting about 5 or 6 years ago, this is our first successful moose hunt.  Bow hunting is a heck of a lot more challenging than rifle hunting, and requires a whole different approach.  With rifle, we used to fill our tags pretty much every year, and there was little challenge left — bow hunting has brought a whole new level of skill to it!

    And if you’re wondering how an arrow in the ass killed a moose — it turns out that it hit just in the right spot to sever a main artery.  There was massive internal bleeding, and he must have dropped pretty fast.  A poor shot turned into a great shot just by pure luck, which is good because it’s infuriating as an ethical hunter if you ever merely wound an animal.

    And now, back out to the hunt camp for more… sitting in a hammock and reading.   😉


  • Our New Wooden Beauty

    My first “real” post on this blog is going to show my gamer-geek side.  Today, my girlfriend and I took delivery of our long-awaited (and totally worth the wait!) piece of gaming kit:  a Geek Chic “The Portal” gaming table.  Geek Chic has been awesome throughout the (long) process of ordering, manufacturing, and delivering this thing… kudos to the entire crew there!  This is some seriously high-quality furniture — I’m very impressed with the build quality.   Can’t wait to play some D&D on this sucker!

    And now, for some eye candy.

    Here’s a picture of the table in “dining table” configuration.  Looks just like a regular dining table, right?

     

    Now we peel off the 5 leaves that comprise the table top and voila — we reveal a recessed gaming surface!  Geek Chic calls this area the “vault” of the table.  The vault on our table is 2.25 inches deep.   To give you an idea of the size of this table, the INSIDE gaming surface shown here is 4×6 feet!

     

    Looking a bit closer at the vault, you can see that the interior is definitely designed for gaming!  What you see here is actually 3 layers — the “floor” of the vault is a sheet of bamboo (which is an upgrade we bought — the standard is just white), above that is a pre-printed sheet with a 1-inch grid (perfect for D&D!  This was also an addon that we bought), and above that is an acrylic sheet, which protects everything underneath (and is safe to use with dry-erase markers!).

     

    The neat thing about this table is that it’s modular.  If you want to wage epicly huge battles (say, for Warhammer or BattleTech), just take the leaves off and you have a 4×6-foot playing surface.  However, most games don’t need quite that amount of room, and it’s nice to have room to put other things, like dice bags, source books or snacks!    By purchasing the “leaf desk” upgrade kit, you can add a “border” around your table, shrinking the size of the Vault and giving your players a bit more room to work.

     

    However, we’re not done yet!  One final feature of this table (and all of Geek Chic’s tables) is that they have a modular rail system built into the sideboards.  You can add things like little “desks”, cup holders, bins, counter/dice holders… you can even buy an “entropy engine” (i.e. dice tower) that sits in the modular rails!

    Of course, we sprung for a bunch of these upgrades as well.  At the edge nearest to the camera, you can see a desk with a shelf underneath it — this one is meant for the GM! — and a burgundy-lined bin and a cup holder.  On the right side, you can see 2 player desks (without a shelf underneath) and a counter holder.  We have more desks and bins/holders on the other 2 sides as well!

     

    The quality of craftmanship in this thing is pretty impressive.   The delivery guys (who have been on a week-long trek across the northern USA and some parts of Canada delivering 18 of these things!) said that each table is built buy a single craftsman from start to finish.  From cutting the lumber to size right through to final assembly, each table is only touched by 1 person until it hits the delivery crew.  This process lets that single person have absolute control over every detail of your table, giving you a higher quality product than just assembly-line style construction.

    The delivery guys are great too — they’re far more than just “delivery”.  They assemble the whole thing for you in situ and stick around to show you how to use (how to insert/remove accessories, leaves, etc) and care for your table properly (even going into details like which wood wax to use!).

    And I can’t forget the support staff, for example the “Valet” who helps you place your order.  They’re very knowledgeable in helping you pick out things like “which accessories should I get if I play game X?”, as well as keeping your apprised of the progress of your new wooden beauty throughout the whole process (including sending you in-progress pictures!).

    In fact… let me just cut this short by saying that the whole crew at Geek Chic are great.

    I’m totally excited to have this table (even if it does mean reorganizing that room now, because as you can see, we don’t have room for chairs around it right now!).   I’m sure it will soon be the envy of our entire gaming group.  😉


  • Howdy!

    This is the home of… well, me.  There will be content here soon, I promise!

    Expect to find random info about all topics sysadmin (storage, networking, servers, operating systems), open-source projects I’m working on, and other random interests (guns, gaming, food and photography)!