Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In with OpenID
Sign In Register
Advertise on LowEndTalk.com

Categories

In this Discussion

› › rclone now supports Hubic

rclone now supports Hubic

amhoabamhoab Member
in General

rclone is a cli utility that allows you to interact with multiple storage backends. I just saw that they released a new version that supports Hubic. I've done minimal testing, but it appears to work fine so far. I figured this would be of use to many of you here.

Downloads: http://rclone.org/downloads/

Github: https://github.com/ncw/rclone

Thanked by 5ATHK tommy Baris inthecloudblog WebProject

Comments

  • MarkTurnerMarkTurner Member, Provider

    Whats performance like outside of OVH's network?

    $10 off your first month's dedicated server using coupon 'SHIPPINGLABEL'
    https://cc.delimiter.com/cart/dedicated-servers/

  • amhoabamhoab Member
    edited November 2015

    I copied a couple of files, 100MB and 500MB each, from a VPS I have in London. The average throughput was around 50Mbps, although it varied across runs.

    # rclone copy 100mb.test hubic:default/ -v
2015/11/16 07:01:19 Hubic: Got swift credentials (expiry 2015-11-17 04:27:58 +0100 +0100 in 23h26m38.277154845s)
2015/11/16 07:01:19 Hubic Swift container default: Modify window is 1s
2015/11/16 07:01:19 Hubic Swift container default: Building file list
2015/11/16 07:01:20 Hubic Swift container default: Waiting for checks to finish
2015/11/16 07:01:20 Hubic Swift container default: Waiting for transfers to finish
2015/11/16 07:01:35 100mb.test: Copied (new)

Transferred:    104857600 Bytes (6349.81 kByte/s)
Errors:                 0
Checks:                 0
Transferred:            1
Elapsed time:       16.1s

2015/11/16 07:01:35 Go routines at exit 27
  • MarkTurnerMarkTurner Member, Provider

    Thats not bad

    $10 off your first month's dedicated server using coupon 'SHIPPINGLABEL'
    https://cc.delimiter.com/cart/dedicated-servers/

  • ATHKATHK Member

    Love rclone great rsync like cloud syncing tool.

  • MarkTurnerMarkTurner Member, Provider

    I finally got a tutorial for RClone up on Delimiter's wiki. I've been playing with RClone for the past few days. Kudos to @amhoab for the recommendation.

    http://wiki.delimiter.com/display/DEL/ObjSpace:+Sync+with+RClone

    So you can try it properly, there's a free trial of ObjSpace 100GB plan for a month and see how you like it. Its $2.50/month thereafter if you keep it, you can scale up if you need more space.

    http://www.delimiter.com/landingpage/objspace/

    Coupon: 1HD16WH7JS

    $10 off your first month's dedicated server using coupon 'SHIPPINGLABEL'
    https://cc.delimiter.com/cart/dedicated-servers/

    Thanked by 1erkin
  • bohdansbohdans Member

    @MarkTurner did the ObjStorage prices drop in the last few days? I am sure 100GB was $10 last time I looked.

  • MarkTurnerMarkTurner Member, Provider

    bohdans said: did the ObjStorage prices drop in the last few days? I am sure 100GB was $10 last time I looked.

    100GB is a new entry point, previously the lowest entry point was 500GB for $9.80

    $10 off your first month's dedicated server using coupon 'SHIPPINGLABEL'
    https://cc.delimiter.com/cart/dedicated-servers/

    Thanked by 1bohdans
  • inthecloudbloginthecloudblog Member
    edited November 2015

    @amhoab said: I copied a couple of files, 100MB and 500MB each, from a VPS I have in London. The average throughput was around 50Mbps, although it varied across runs.

    > # rclone copy 100mb.test hubic:default/ -v
> 2015/11/16 07:01:19 Hubic: Got swift credentials (expiry 2015-11-17 04:27:58 +0100 +0100 in 23h26m38.277154845s)
> 2015/11/16 07:01:19 Hubic Swift container default: Modify window is 1s
> 2015/11/16 07:01:19 Hubic Swift container default: Building file list
> 2015/11/16 07:01:20 Hubic Swift container default: Waiting for checks to finish
> 2015/11/16 07:01:20 Hubic Swift container default: Waiting for transfers to finish
> 2015/11/16 07:01:35 100mb.test: Copied (new)
> 
> Transferred:    104857600 Bytes (6349.81 kByte/s)
> Errors:                 0
> Checks:                 0
> Transferred:            1
> Elapsed time:       16.1s
> 
> 2015/11/16 07:01:35 Go routines at exit 27
>

    Which vps to Hubic?

    inthecloudblog.net

  • bohdansbohdans Member

    @markturner I like the domain for the ObjStorage

    Thanked by 1MarkTurner
  • tommytommy Member

    from Leaseweb SG to Hubic, I can max out my upload speed

    Transferred:   1000000000 Bytes (8803.78 kByte/s)
Errors:                 0
Checks:                 0
Transferred:            1
Elapsed time:     1m50.9s

    test upload 954 M

    Let's bet which dot-name will collapse first ;). Super Hot VPS Deal

    Thanked by 1vimalware
  • TinyTunnel_TomTinyTunnel_Tom Member

    OVH>Online.net is horrid. 40Gbit of connectivity on somedays 2 of the links are at 97% the other 2 are at 92%

    Different.

  • rmlhhdrmlhhd Member
    edited November 2015

    @MarkTurner said:

    So you can try it properly, there's a free trial of ObjSpace 100GB plan for a month and see how you like it. Its $2.50/month thereafter if you keep it, you can scale up if you need more space Coupon: 1HD16WH7JS

    Shame you only have a US location for object storage

    RIPE NCC Member

  • MarkTurnerMarkTurner Member, Provider

    rmlhhd said: Shame you only have a US location for object storage

    If there is demand, then we could deploy in any of our EU sites, object storage is sufficiently profitable to operate in EU and Asia.

    $10 off your first month's dedicated server using coupon 'SHIPPINGLABEL'
    https://cc.delimiter.com/cart/dedicated-servers/

  • NihimNihim Member
    edited November 2015

    Can someone check how fast files are deleted? If I remember when I got my hubic acc people were saying files too a long as time to get removed.

    Also using some hubic api at that time, any file uploaded had to be copied first to some special partition or dir of the api (which was quite a waste of I/O).

    Does that still happen with the above method?

    ps: obviously I never did manage to test for myself even though my hubic 10T acc is active and paid since their offer back then :/

  • mosan7763mosan7763 Member
    edited November 2015

    any guide to set it up on debian? when following the linux guide i just get stuck at "waiting for code..".

    the webpage reports: "Failed! No code found." (I access it over ssh tunnel)...

  • CFarenceCFarence Member

    @MarkTurner said: Whats performance like outside of OVH's network?

    I just transferred 800MB from Delimiter to Hubic and got 5056.04 kByte/s.

    Thanked by 2MarkTurner vimalware
  • NihimNihim Member

    question: rclone site says that you need to open a browser so it gets the hubic token.

    What if you are running cli?

  • BochiBochi Member

    @Nihim said: question: rclone site says that you need to open a browser so it gets the hubic token.

    What if you are running cli?

    No problem at all, just tested it myself: Only thing you need to do is forwarding a port, then you can go through the usual authentication process of hubiC.

    Thanked by 1Nihim
  • NihimNihim Member

    @Bochi said: No problem at all, just tested it myself: Only thing you need to do is forwarding a port, then you can go through the usual authentication process of hubiC.

    Thanks. As I said I haven't tried it myself so not quite sure what you did. If you feel like it you could drop a few lines of noob-proof steps.

  • mosan7763mosan7763 Member

    somehow the upload speed from online.net with rclone is only 7mbit. I usually got 150-200mbit with hubiCs own app and about 100mbit with hubicfuse. strange.

  • BochiBochi Member
    edited November 2015

    @Nihim said: If you feel like it you could drop a few lines of noob-proof steps.

    Sure! Just answered a question regarding this by a private message...

    For example with Putty (ssh on linux should achieve the same): It is as simple as creating a tunnel to "127.0.0.1:53682", then you can access the remote path on your local source port and go through the normal App-Login mechanism of hubiC.
    Client ID and secret will then be stored in your local config file afterwards and be used to access your storage.

    /EDIT/
    But as far as I can judge that after this short time it seems as if I get slower uploads as for example with duply or their original client.

    Thanked by 1ATHK
  • KupolKupol Member

    @mosan7763

    online.net's links to OVH are overloaded most of the time. This could cause slow speed to OVH from online.net.

  • amhoabamhoab Member

    @Nihim said: question: rclone site says that you need to open a browser so it gets the hubic token.

    What if you are running cli?

    I just opened another shell with tmux and used lynx to open that URL. rclone was able to detect that, and updated the config automatically.

  • amhoabamhoab Member

    @inthecloudblog said: Which vps to Hubic?

    This was from a We Love Servers VM in London.

    Thanked by 1inthecloudblog
  • mosan7763mosan7763 Member
    edited November 2015

    only problem i have with rclone is that the transfer speed is always ~800kbyte/s per file. it doesn't matter how many (1 or 20) it's always ~800kbyte/s.

  • MarkTurnerMarkTurner Member, Provider

    mosan7763 said: only problem i have with rclone is that the transfer speed is always

    Rclone to what?

    $10 off your first month's dedicated server using coupon 'SHIPPINGLABEL'
    https://cc.delimiter.com/cart/dedicated-servers/

  • mosan7763mosan7763 Member

    hubiC

  • MarkTurnerMarkTurner Member, Provider

    @mosan7763 - Thats more likely to be hubic/network than Rclone. Quite a few people complain about performance of hubic even with the hubic client.

    I get almost wirespeed from my test server to Delimiter ObjSpace using Rclone so the software seems to be perfect.

    $10 off your first month's dedicated server using coupon 'SHIPPINGLABEL'
    https://cc.delimiter.com/cart/dedicated-servers/

  • mosan7763mosan7763 Member

    a simple trick is to just queue more files.

  • NeoonNeoon Member

    Hubic is limited to 1MB/s, so 800kb/s is pretty close.

    MyLowEndBoxes Status - Free Backupspace for your Lowendspirit Box

  • chrispchrisp Member
    edited November 2015

    Kupol said: online.net's links to OVH are overloaded most of the time. This could cause slow speed to OVH from online.net.

    I am backing things up from Online.net to Hubic on a regular basis and I saturate the full gigabit uplink with only one connection. It's crazy how fast it is.

    Infinity580 said: Hubic is limited to 1MB/s, so 800kb/s is pretty close.

    Definitely not true. Via API (hubicfuse) I get 114MByte/s. Uploaded almost 7TiB in the last 2 months.

    Thanked by 1inthecloudblog
  • NihimNihim Member
    edited November 2015

    Well the bad thing with the https is that the overhead (n = 4) is killing my weak ass cpu (via nano). Otherwise while using a lot of I/O for other stuff, for one file it reached an avg of 2.5MB/s

    Something I just saw. Only the contents of default dir are visible when you are logged into the hubic web account? Any way to modify that?

  • CFarenceCFarence Member

    @Nihim said: Any way to modify that?

    From what I read on the hubic forums, no you can't modify it. Though you can use SwiftExplorer to see other containers.

  • NihimNihim Member

    @CFarence :
    Thanks rclone's ls will have to do. Atm I am stuck on how the hell do I remove a file from hubic. I don't see any remove option outside sync

  • CFarenceCFarence Member

    @Nihim said: CFarence :
    Thanks rclone's ls will have to do. Atm I am stuck on how the hell do I remove a file from hubic. I don't see any remove option outside sync

    Yeah, I had the same issue, finally used SwiftExplorer to empty it out. I wish the web interface would let you change containers. It'll tell you usage on the account, but can't view it

  • NihimNihim Member

    with rclone from NL using default n=4 got 50MB/s up & 60MB/s down

    So the big problem is removing files and having to manually? refresh the token

  • amhoabamhoab Member

    @Nihim said: CFarence :
    Thanks rclone's ls will have to do. Atm I am stuck on how the hell do I remove a file from hubic. I don't see any remove option outside sync

    To remove a file or dir, rclone purge <path> works for me.

    Thanked by 3Nihim CFarence baran
  • NihimNihim Member

    @amhoab
    nice! I thought it didn't work when I tried it cause apparently lsd container is slow to update even though if you do ls container the file is removed.

  • amhoabamhoab Member

    @Nihim said: amhoab
    nice! I thought it didn't work when I tried it cause apparently lsd container is slow to update even though if you do ls container the file is removed.

    I've seen that before as well; the deletes seem to happen async, but I don't think that's an rclone thing, but just how Hubic/Swift work.

  • NihimNihim Member

    yea it's hubic, can't blame the tool used. Seems to me though the file deletion is pretty fast, just container info is slow to update.

  • squibssquibs Member

    Hmm. I've set this up and run my sync, and all appears well when I run rsync ls or rclone size. However, I can't see the sync data on the hubic web portal - should I be able to? I can see the rclone app registered in "my account->your applications."

    Ask me about my experiences with Wable.

  • amhoabamhoab Member

    If you upload to anything but the default container, you won't see it in the UI. Which container did you upload to?

  • squibssquibs Member

    Can't remember.

    Ask me about my experiences with Wable.

  • BochiBochi Member

    @squibs said: Can't remember.

    Then use something like Swift Explorer so see the other containers.

    Thanked by 1squibs
  • squibssquibs Member

    @Bochi said: Then use something like Swift Explorer so see the other containers.

    Nifty - ta!

    Ask me about my experiences with Wable.

Sign In or Register to comment.