Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Advertise on LowEndTalk.com
Recommendation for serving files stored on VPS through a web index page hosted on another server?
New on LowEndTalk? Please Register and read our Community Rules.

Recommendation for serving files stored on VPS through a web index page hosted on another server?

Can LET recommend some scripts or ways to create a pooled index of stored on multiple servers and serve those files through a single http page? I remember having a free script that the creator had published many years ago that did this, but don't remember what it was called. Basically, it could be run in a server mode on the main server that served the index (and it could also locally have files to be served) or in a client mode on the other servers that were hosting files referencing the folder holding the files to be served. The script would generate an up-to-date pooled open directory-type index listing all of the files/folders on all of the servers and clicking to download any of them would serve the appropriate file. I don't remember for sure if it routed all downloads through the webserver, but think it may have.

E.g. Server A is a server with a small HDD, lots of BW, and a running webserver; Servers B-D are VPS with larger HDDs that have no webserver, but contain the files to be served. The goal is to have an up-to-date file index webpage accessible on Server A (e.g. https://domain.pointing.to.servera/mycrap.html) that is a complete index of the files I want to be served from Servers B-D such that I can download the files by browsing the index folders and clicking the file links. There is some flexibility as to whether Server A is a shared hosting account (with or without SSH access) or a VPS, though it would be great to be able to support both options.

Personal consultant to OP's Mom™

Comments

  • I found the program I had used in the past: https://github.com/null-dev/WebLinkedServer

    However, it was last updated in 2017, so any more up to date alternative or one that supports Server A not having SSH access would be great.

    Personal consultant to OP's Mom™

  • dahartigandahartigan Member without signature
    edited November 23

    You could just create symlinks to your files directories in your www root, if you use sshfs/rclone symlinks to the mount points and you're giggling.

    Edit this is not going to work on shared hosting simply because you can't mount an external filesystem like that.

    Thanked by 1user123
  • @dahartigan said:
    Edit this is not going to work on shared hosting simply because you can't mount an external filesystem like that.

    I'm curious if there is some way set up a cron job on the shared hosting account that connects to other servers to update an index which it can then serve.

    Personal consultant to OP's Mom™

  • dahartigandahartigan Member without signature

    This is where it gets messy, and where I admit I don't fully understand your intentions here. Do you want a list of links or do you want those files "tunnelled" through the site?

    Please clarify ;)

  • @dahartigan said:
    This is where it gets messy, and where I admit I don't fully understand your intentions here. Do you want a list of links or do you want those files "tunnelled" through the site?

    Please clarify ;)

    I think the latter would be required, since Servers B-D would not have webservers (or at least not have those files in a publicly-accessible webfolder), though they would have SSH/SFTP or any other script/app that can be installed on a VPS.

    Personal consultant to OP's Mom™

  • dahartigandahartigan Member without signature

    @user123 said:

    @dahartigan said:
    This is where it gets messy, and where I admit I don't fully understand your intentions here. Do you want a list of links or do you want those files "tunnelled" through the site?

    Please clarify ;)

    I think the latter would be required, since Servers B-D would not have webservers (or at least not have those files in a publicly-accessible webfolder), though they would have SSH/SFTP or any other script/app that can be installed on a VPS.

    Okay well shared hosting would be a no-go for that :/

  • @dahartigan said:

    @user123 said:

    @dahartigan said:
    This is where it gets messy, and where I admit I don't fully understand your intentions here. Do you want a list of links or do you want those files "tunnelled" through the site?

    Please clarify ;)

    I think the latter would be required, since Servers B-D would not have webservers (or at least not have those files in a publicly-accessible webfolder), though they would have SSH/SFTP or any other script/app that can be installed on a VPS.

    Okay well shared hosting would be a no-go for that :/

    Darn. So even something like a hidden PHP/ajax terminal script wouldn't work?

    Personal consultant to OP's Mom™

  • @dahartigan said:

    @user123 said:

    @dahartigan said:
    This is where it gets messy, and where I admit I don't fully understand your intentions here. Do you want a list of links or do you want those files "tunnelled" through the site?

    Please clarify ;)

    I think the latter would be required, since Servers B-D would not have webservers (or at least not have those files in a publicly-accessible webfolder), though they would have SSH/SFTP or any other script/app that can be installed on a VPS.

    Okay well shared hosting would be a no-go for that :/

    In that case, what if Server A is a VPS with a webserver and I want the files from the other servers (that don't have webservers) tunneled/served through it?

    Thanked by 1dahartigan

    Personal consultant to OP's Mom™

  • dahartigandahartigan Member without signature

    @user123 said:

    @dahartigan said:

    @user123 said:

    @dahartigan said:
    This is where it gets messy, and where I admit I don't fully understand your intentions here. Do you want a list of links or do you want those files "tunnelled" through the site?

    Please clarify ;)

    I think the latter would be required, since Servers B-D would not have webservers (or at least not have those files in a publicly-accessible webfolder), though they would have SSH/SFTP or any other script/app that can be installed on a VPS.

    Okay well shared hosting would be a no-go for that :/

    In that case, what if Server A is a VPS with a webserver and I want the files from the other servers (that don't have webservers) tunneled/served through it?

    This you can do!

    1) On the VPS A install rclone and set up all of your "remote servers"
    2) mount those somewhere outside of the var www directory
    3) create symlinks in var www html that point to those mounts
    4) visit VPS a and connect to a symlink, if you want to list them delete the index html file in var www html.

    Thanked by 1user123
  • farsighterfarsighter Member
    edited November 23

    Check rclone + mergerFS.

  • I just tried this using gdrive and rclone. Works great. No more storing things locally for me. :)

    Thanked by 3dahartigan user123 MrH
  • @Barnesanger said:
    I just tried this using gdrive and rclone. Works great. No more storing things locally for me. :)

    Is there a guide or walkthrough you would recommend?

    Personal consultant to OP's Mom™

  • dahartigandahartigan Member without signature

    @user123 said:

    @Barnesanger said:
    I just tried this using gdrive and rclone. Works great. No more storing things locally for me. :)

    Is there a guide or walkthrough you would recommend?

    Official doc: https://rclone.org/drive/

    May look scary but it's really not. PM if you get stuck and I'll hold your hand, but just don't get frisky..

    Thanked by 1user123
  • try minio

  • @dahartigan can you mount SFTP drives from shared hosting if you have jailed SSH access?

    Personal consultant to OP's Mom™

  • dahartigandahartigan Member without signature
    edited November 26

    @user123 said:
    @dahartigan can you mount SFTP drives from shared hosting if you have jailed SSH access?

    Nope because you need access to a kernel module called fuse, which means you need to generally be the root user.

    Edit: Just get a cheap little VPS ya tightwad haha it'll 100% work

    Thanked by 1user123
  • @dahartigan said:

    @user123 said:
    @dahartigan can you mount SFTP drives from shared hosting if you have jailed SSH access?

    Nope because you need access to a kernel module called fuse, which means you need to generally be the root user.

    Edit: Just get a cheap little VPS ya tightwad haha it'll 100% work

    Lmk if you come across anything super cheap with high BW :)

    Personal consultant to OP's Mom™

  • dahartigandahartigan Member without signature
    edited November 26

    @user123 said:

    @dahartigan said:

    @user123 said:
    @dahartigan can you mount SFTP drives from shared hosting if you have jailed SSH access?

    Nope because you need access to a kernel module called fuse, which means you need to generally be the root user.

    Edit: Just get a cheap little VPS ya tightwad haha it'll 100% work

    Lmk if you come across anything super cheap with high BW :)

    Define high bandwidth

    Edit: and it's not going to be a public file server is it? Because that would mean you'd need DMCA free. If it's just for you and your mates it's not likely to matter.

  • user123user123 Member
    edited November 26

    @dahartigan said:

    @user123 said:

    @dahartigan said:

    @user123 said:
    @dahartigan can you mount SFTP drives from shared hosting if you have jailed SSH access?

    Nope because you need access to a kernel module called fuse, which means you need to generally be the root user.

    Edit: Just get a cheap little VPS ya tightwad haha it'll 100% work

    Lmk if you come across anything super cheap with high BW :)

    Define high bandwidth

    Edit: and it's not going to be a public file server is it? Because that would mean you'd need DMCA free. If it's just for you and your mates it's not likely to matter.

    I was vague because I don't really have a threshold. I don't think I've hit even the 500GB BW threshold on a VPS, but tunneling will double the amount of bandwidth used and it would be to tunnel to my 500GB HDD VPS. So, probably a couple TB just to be safe? DMCA is irrelevant and nothing would be public.

    Personal consultant to OP's Mom™

Sign In or Register to comment.