Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


How transfer very big site?
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

How transfer very big site?

hello,

i need to change server for a site of 250gb
II can't do any backup even on extarnal server because there isn't enough space to do that.
Plesk migrator not work
I try to download by ftp but it's very slow and not download all files

how can i move?

«1

Comments

  • rsync is your friend.

    Thanked by 2Shazan redalertrox
  • desperanddesperand Member
    edited August 2018
    1. Buy any cloud storage VPS with enough amount of space for a time while you're moving your site to a new space
    2. Move all data there to the VPS (for example: https://www.vultr.com/pricing/blockstorage/)
    3. buy a new server where you will be hosted
    4. configure everything
    5. move from vultr VPS (just example, you can use anything) via SCP / Rsync / rclone / anything files back to your main VPS / Dedi
    6. change dns
    7. profit???

    ===============

    Also, personally me prefer a b2 because of the price.
    Moving (download / upload / store) 250GB of data will be cost around 3-4$ with blackbaze. Wich is pretty nice.

    https://www.backblaze.com/b2/cloud-storage-pricing.html

    What will I do in your case?

    1. Setup any backup solution (duply / rclone / forgot name), personally me likes duply
    2. duply backupname backup
    3. then will setup a new hosting server and configure nginx / php / so on
    4. then will move config to a new server of duply and will execute duply backupname restore /folder/where/site/will/be/hosted
    5. will wait while whole site will be moved
    6. change DNS
    7. profit??
    Thanked by 2chiccorosso NanoG6
  • edited August 2018

    desperand said: Move all data there to the VPS (for example: https://www.vultr.com/pricing/blockstorage/)

    desperand said: duply backupname backup

    He will definitely fail at your second steps.

  • work with plesk?

  • jsgjsg Member, Resident Benchmarker

    Assuming that "site" means some directory/ies and everything in them you can do this (on the "old" server:

    • tar the directory/ies to be transfered ...
    • ... with some good compression and have the tar output piped to ...
    • ssh [email protected] 'cat > oldserver-backup.tar.bz2'

    Notes:

    Thanked by 1vimalware
  • jsgjsg Member, Resident Benchmarker
    • It is assumed that you have ssh running on the new server
    • my example assumes bzip2 compression (tar cj ...); change filename if you use another scheme.
  • jsgjsg Member, Resident Benchmarker

    I had prepared a full example but %%§$"/§ Cloudflare is f*cking up everything. I've had enough. Sorry.

    Thanked by 2NanoG6 sin
  • ShazanShazan Member, Host Rep

    I would just rsync it...

  • edited August 2018

    Shazan said: I would just rsync it...

    The problem is that he doesn't have enough space and most of the comments above suggest create bzip/tar files and move it...

    @chiccorosso said:
    work with plesk?

    You don't have enough space to create backup so you can't move the whole settings to the new server.
    You will have to create per site settings for new site, then rsync the content and move db to the new server.

    Another choice is asking if your current provider provide local renting and mounting another disk on it. Then mount the drive, do the backup and move it to the new server.

  • Use rsync to create identical copy ot the site/server elswhere.
    If it contains only static files, then, it will be no issue. If there are mysql databases also, there are many routes you can follow.

    For example, instead of creating backup of the entire site, create only mysql dumbs. Then, copy static files and restore the mysql databases to the new server.

    You haven't told us what kindy of infrastructure you have. Is it a website like Wordpress or Joomla? If yes, another trick you can follow to create backup via app (e.g. akeeba backup) and you have not enough space for that on the initial server, is to move via rsync certain folders that usually takes all the space (e.g. images and videos) to the backup server. This way, you gain a lot of space in the initial node. Do the backup via app, move the site and then, restore to the original place in the new node the moved static files (images and videos).

  • which type of website. which consume 250 gb.
    it is video type website.

  • jsgjsg Member, Resident Benchmarker

    @khuongcomputer said:
    You don't have enough space to create backup so you can't move the whole settings to the new server.

    No. Some solutions (like mine) need (almost) no space but transmit the output directly to the new server.

    The reason I prefer my solution over some others (e.g. rsync) is that it's simple, secure, and COMPRESSES - which you definitely want when transferring 250 GB.

    In fact I myself don't use tars "built-in" compression but (properly parametrized) zstd because compression is probably the single most important factor in that kind of use case but I wanted to keep it simple for OP and e.g. bzip2 isn't too bad.

  • @jsg said:

    @khuongcomputer said:
    You don't have enough space to create backup so you can't move the whole settings to the new server.

    No. Some solutions (like mine) need (almost) no space but transmit the output directly to the new server.

    The reason I prefer my solution over some others (e.g. rsync) is that it's simple, secure, and COMPRESSES - which you definitely want when transferring 250 GB.

    rsync over ssh (rsync -avze ssh /bla root@destination:/bla) is also encrypted and compressed. And it won't transfer things didn't change (unlike tar).

    Thanked by 1Shazan
  • ShazanShazan Member, Host Rep

    @khuongcomputer said:

    Shazan said: I would just rsync it...

    The problem is that he doesn't have enough space and most of the comments above suggest create bzip/tar files and move it...

    Space for what? Rsync doesn't need additional space on the source server and can transfer files compressed and encrypted.

  • Shazan said: Space for what? Rsync doesn't need additional space on the source server and can transfer files compressed and encrypted.

    Re-read my whole sentence please?

  • oh boy.. this is going to be complex.. and I am not talking about moving the site. ;)

    Thanked by 2imok Falzo
  • @khuongcomputer said:

    Shazan said: Space for what? Rsync doesn't need additional space on the source server and can transfer files compressed and encrypted.

    Re-read my whole sentence please?

    You can directly pipe tar over ssh. It won't use any space on the source.

  • @mhosting_in said:
    which type of website. which consume 250 gb.
    it is video type website.

    That is actually interesting. If you could deliver some specific infos @chiccorosso

    There can be different backup/migrate strategies depending on your case.

  • no need to compression you can use file copy and transfer with rsync

  • At low hardware compression may be worsen copy

  • @jsg said:
    I had prepared a full example but %%§$"/§ Cloudflare is f*cking up everything. I've had enough. Sorry.

    You could just use Pastebin for those

  • The thing here is that rsync alone won't transfer mysql or other databases (at least, correctly). He needs to dumb them. Also, we don't know if he just want to copy a website or all the infrastructure (e.g. web panel, settings, special configs etc.).
    OP has disappeared. Either he found out what to do, or he dont need it anymore (I guess).

  • ShazanShazan Member, Host Rep

    @jvnadr said:
    The thing here is that rsync alone won't transfer mysql or other databases (at least, correctly). He needs to dumb them.

    You could rsync then stop all the services and rsync everything again. This way the dbs and other open files are transferred correctly.

  • @chiccorosso said:
    hello,

    i need to change server for a site of 250gb
    II can't do any backup even on extarnal server because there isn't enough space to do that.
    Plesk migrator not work
    I try to download by ftp but it's very slow and not download all files

    how can i move?

    Depending on what makes up that 250GB of data, you could do it in stages

    1. If some of the data is made up of individually large files i.e. zip, tar.gz, video or other megabyte/gigabyte sized files, then you can start moving the largest of those files over to another server via rsync as folks have mentioned. For example if you had 50x 1GB sized files = 50GB of data can be rsync'd over to another server and then removed from the existing server. This frees up 50GB of disk space for you.

    2. If freeing up largest files via rsync move gives you enough free disk space now, you can use whatever means to transfer the rest of your data normally :)

  • jsgjsg Member, Resident Benchmarker

    @jvnadr said:
    The thing here is that rsync alone won't transfer mysql or other databases (at least, correctly). He needs to dumb them. ...

    There is ways/tool for any halfway decent SQL server to dump tables and/or database(s) to stdout which can then be piped into compression and then into ssh.

    The real problem is integrity incl. the question what happens to data that have been entered after the transfer. One solution is to simply put the DB in read only mode (web site works except no new entries) and dump it.

    If one prefers rsync and the DB isn't too big a multi phase process like the one suggested by @eva2000 might work but put the web site out of order somewhat longer. Like: first transfer a lot of non-DB stuff to make room for the DB dump.

  • Sounds like OP is using Windows and Plesk. Does rsync and ssh work in Windows? Genuine question as I haven't touched Windows in a decade or more.

  • eva2000eva2000 Veteran
    edited August 2018

    jsg said: There is ways/tool for any halfway decent SQL server to dump tables and/or database(s) to stdout which can then be piped into compression and then into ssh.

    Yeah one method (once you have freed up some disk space first) would be to use mydumper for parallel multi-threaded database backup for individual tables to sql files https://github.com/maxbube/mydumper and then instead of normal compression methods like gzip, you could use facebook's zstd https://github.com/facebook/zstd with data dictionary training which can improve zstd compression ratios by 3.5x so you end up with much smaller individual table sql compressed files which you can rsync across to another server.

    Or even without zstd data dictionary training there are other compression tools which much better compression ratios which result in smaller compressed file sizes. I benchmarked various compression tools including ztsd vs gzip vs pigz vs brotli etc at https://community.centminmod.com/threads/round-2-compression-comparison-benchmarks-zstd-vs-brotli-vs-pigz-vs-bzip2-vs-xz-etc.15126/. This is all provided you first free up some disk space by transferring the largest individual files first via rsync and removing their local copies on existing server. Example compression ratio for uncompressed to compressed size for gzip level 9 at 3.0767 vs for lbzip2 level 9 at 3.8784 vs pxz level 6 at 4.2768 vs zstd level 19 at 3.9695 vs brotli level 6 at 3.6224

    Also log files are good candidate for compression to free up disk space if you still need to keep the log files. If you do not need the log files, just delete them.

  • 2gouzi2gouzi Member
    edited August 2018

    scp -r -P 22 root@IP:/home/old/ /home/new

    Quote: http://www.zxsdw.com/index.php/archives/1138/

  • jsgjsg Member, Resident Benchmarker

    @2gouzi said:
    scp -r -P 22 root@IP:/home/old/ /home/new

    Quote: http://www.zxsdw.com/index.php/archives/1138/

    Bad idea. For one without -C flag there is no (gzip) compression. More importantly though the -r flag follows symbolic links.

Sign In or Register to comment.