Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In with OpenID
Advertise on LowEndTalk.com

In this Discussion

Rsync out of memory? Try this...

Rsync out of memory? Try this...

pubcrawlerpubcrawler Banned
edited December 2012 in Tutorials

If you do proper backups and are a shell type person, you probably are an rsync user.

In the lowend market, where there is limited RAM available, rsync can really fail in a big way. Failure is out of memory related when rsync attempts to load all the filenames and info in to RAM at startup.

If you are getting out of memory errors and done the search engine researching and swear rsync has a bug, fear not.

This is your new rsync commandline option to postpend to your current script or cron job:

--no-inc-recursive

That cured our ills on a 256MB VPS. Bound to work just as well on smaller instances.

Tagged:
Thanked by 1cheapdave

Comments

  • Nice tip!

    :) I am available for hire as a systems administrator, customer support technician levels 1-3, including managed support. Based in UK. Flexible time-zones and hours.

  • pubcrawlerpubcrawler Banned
    edited December 2012

    I run into this issue all the time on smaller RAM ARM boxes. Haven't given it a spin yet on those. The feature was released somewhere in the 3.xxx release timeline, so it's "recent" and doesn't appear in the ancient write ups folks have on rsync from 2005 :)

  • Nice tip, will be using that on my new storage box, thanks.

  • Working well on my RPi, big improvement actually.

    Do not click this link.
  • corpuscorpus Member
    edited December 2012

    Incremental recursion can be disabled using the --no-inc-recursive option or its shorter --no-i-r alias.

    Nice tip

    apt-get -a beer

  • Glad it helps @Andre!

    Thanks @corpus. I like your tag line there! Too bad apt-get won't a beer :)

  • @pubcrawler said: Too bad apt-get won't a beer :)

    It won't if you're not running the command as superuser. (-:

    Relevant xkcd

    Do not click this link.
  • @andre I just had to justify what sudo meant in class... Because I was in piss fits at that.

    Junior Pen Tester

  • You can also use something like,

    --rsync-path="nice -n 19 /usr/bin/rsync"

    To lower the priority of the resync process on the remote end. Apparently, with a modern kernel nice also adjusts IO priority.

    Of course you'd also nice the local end with,

    nice -n 19 /usr/bin/rsync ...

  • Thanks @pubcrawler, @sleddog. Very nice. :-)

    @Andre - lol

    -

Sign In or Register to comment.