Page 1 of 2 12 LastLast
Results 1 to 25 of 39

Computer Nerds - Short term / High Volume Cloud Storage

  1. #1
    Senior Member
    Join Date
    Jan 2014
    Location
    Framingham, MA
    Posts
    523
    iTrader Score
    0

    Computer Nerds - Short term / High Volume Cloud Storage

    So I've got a ZFS pool of four 3tb disks in my home freenas box and I can't believe it but we filled the thing up. I didn't do my research and missed that we're only supposed to have 20% of the pool free or risk calamity. Also my wife's full frame raw files are like 50MB each and there's already 3TB+ of her shit (professional photographer). I cut back on my media files but I still have ~5.5TB of files and I want to expand my pool which I cannot do without destroying the current one.

    This leads me to my problem, I need ~5.5TB of short term, reliable, storage space without transfer limits so that I can move it in a reasonable amount of time. I just got FIOS gigabit so my end shouldn't be a problem. Google drive is a lot of money for a month of storage, dropbox only does 1TB, amazon isn't cheap and might only be yearly at that size, etc etc etc. I found a great program to run in a jail to move the files called rclone that interfaces with these common cloud services and also with http/ftp. Any suggestions? Anyone have this kind of storage that I could borrow? I'm trying to avoid buying a 8/10TB drive for $300+ that I won't have a use for afterwards. I'm already mad that I'll have to spend money to store the data and then buy two more 3tb drives.

    If no suggestions then memes will do.

    0 Not allowed!

  2. #2
    Lifer ilikenapalm's Avatar
    Join Date
    Mar 2009
    Location
    Connecticut
    Age
    28
    Posts
    1,751
    iTrader Score
    2 (100%)

    Re: Computer Nerds - Short term / High Volume Cloud Storage

    I didn't realize we were calling porn media files these days.

    2 Not allowed!

  3. #3
    Backwoods lobster boy number9's Avatar
    Join Date
    Aug 2005
    Location
    Waltham, MA
    Posts
    3,139
    iTrader Score
    2 (100%)

    Re: Computer Nerds - Short term / High Volume Cloud Storage

    Backblaze? $5/mo and they say it's unlimited...

    I sell data storage for a living, and I think buying an 8TB drive (or two) is going to be your cheapest option. We sell enterprise stuff; my colleagues swear by Synology and Qnap for home use.

    0 Not allowed!
    '03 Porsche 911, '05 KTM 300 EXC, '08 Ducati HyperMotard 1100S, '14 Honda CRF 250R, '14 Subaru XV Crosstrek

  4. #4
    Super Moderator TheIglu's Avatar
    Join Date
    Apr 2003
    Location
    Royalston, MA
    Age
    37
    Posts
    18,505
    iTrader Score
    2 (100%)

    Re: Computer Nerds - Short term / High Volume Cloud Storage

    I use Synology units on my network and I'm happy as shit with them.

    I actually have a couple out of service HP NAS boxes (big boys, 12 bays) populated with 2TB SAS drives that are laying around unplugged. They could host everything you wanted, easily.

    Problem is, SFTP is slow as molasses, so your transfer rate to me will be pretty weak unless we don't encrypt and go straight FTP, which is a security risk for your data.

    Is it worth a DRZ to you?

    1 Not allowed!
    -Clayton
    2006 Suzuki SV650
    2004 Suzuki Vstrom 650
    2001 Kawasaki ZRX1200R
    1986 Honda Oldwing Medicaide
    1982 Honda CB750F Super Sport

  5. #5
    World Trackday Champion xxaarraa's Avatar
    Join Date
    Apr 2012
    Posts
    1,640
    iTrader Score
    2 (100%)

    Re: Computer Nerds - Short term / High Volume Cloud Storage

    Computer Nerds - Short term / High Volume Cloud Storage-told-wtf-youre-talking-jpg

    0 Not allowed!
    All you need to know about NESR:
    Quote Originally Posted by Ductard View Post
    Personally, I would just call up Shell and see which bike they have specified for Rotella.

  6. #6
    Senior Member
    Join Date
    Jan 2014
    Location
    Framingham, MA
    Posts
    523
    iTrader Score
    0

    Re: Computer Nerds - Short term / High Volume Cloud Storage

    Quote Originally Posted by ilikenapalm View Post
    I didn't realize we were calling porn media files these days.
    What else would you call it? I don't think people download porn any more though, why would you when you can just stream it for free on demand?

    0 Not allowed!

  7. #7
    Senior Member
    Join Date
    Jan 2014
    Location
    Framingham, MA
    Posts
    523
    iTrader Score
    0

    Re: Computer Nerds - Short term / High Volume Cloud Storage

    Quote Originally Posted by number9 View Post
    Backblaze? $5/mo and they say it's unlimited...

    I sell data storage for a living, and I think buying an 8TB drive (or two) is going to be your cheapest option. We sell enterprise stuff; my colleagues swear by Synology and Qnap for home use.
    I will check them out. A month of premium cloud storage should cost a fraction of one 8tb drive which is why I've pretty much ruled that option out. Plus I have no other use for it at this time. If you're curious I've got a supermicro board, i3, and 16tb ecc ram. Freenas is pretty awesome freeware and I use it to host ftp, smb, afp, plexmediaserver, sabnzbd, sonarr, couchpotato, transmission, unifi controller, and maybe more that I've forgotten about.

    0 Not allowed!

  8. #8
    Senior Member
    Join Date
    Jan 2014
    Location
    Framingham, MA
    Posts
    523
    iTrader Score
    0

    Re: Computer Nerds - Short term / High Volume Cloud Storage

    Quote Originally Posted by TheIglu View Post
    I use Synology units on my network and I'm happy as shit with them.

    I actually have a couple out of service HP NAS boxes (big boys, 12 bays) populated with 2TB SAS drives that are laying around unplugged. They could host everything you wanted, easily.

    Problem is, SFTP is slow as molasses, so your transfer rate to me will be pretty weak unless we don't encrypt and go straight FTP, which is a security risk for your data.

    Is it worth a DRZ to you?
    My data is personal but it's not sensitive. I wouldn't think that transferring thousands of pictures and tv shows would put me in a compromised position?

    I sold the shit out of the DRZ, I just didn't want to make an "I told you so" post when I got asking price for it [but now I am so suck it everyone]

    - - - Updated - - -

    Quote Originally Posted by xxaarraa View Post
    Computer Nerds - Short term / High Volume Cloud Storage-told-wtf-youre-talking-jpg
    I was hoping for cats but I'll allow it

    1 Not allowed!

  9. #9
    Super Moderator TheIglu's Avatar
    Join Date
    Apr 2003
    Location
    Royalston, MA
    Age
    37
    Posts
    18,505
    iTrader Score
    2 (100%)

    Re: Computer Nerds - Short term / High Volume Cloud Storage

    Pictures is fine. TV Shows, unless you made them (you some kind of furry or some shit?) is out. Can't have them in a place of bidniss.

    And congrats on the DRZ sale! I was just kidding about it, but figured maybe you just wanted it gone and I'd have some leverage

    0 Not allowed!
    -Clayton
    2006 Suzuki SV650
    2004 Suzuki Vstrom 650
    2001 Kawasaki ZRX1200R
    1986 Honda Oldwing Medicaide
    1982 Honda CB750F Super Sport

  10. #10
    Lifer Kurlon's Avatar
    Join Date
    Jun 2007
    Location
    Waterboro ME
    Age
    40
    Posts
    11,743
    iTrader Score
    0

    Re: Computer Nerds - Short term / High Volume Cloud Storage

    So, first question is are you using zfs compression? If you don't have that on, flip the switch. As long as you don't go heavy handed and stick to lz4 you don't get much of a speed hit (in some cases it's actually quicker) and stuff like RAW files end up taking up much less space. You'll need to remove and replace the files to get them compressed but you can do that with much smaller storage and a little planning. I think there are scripts out there that will automate the process for you in a ZFS env.

    Check out rsync.net, and Backblaze for online dumping ground options. If you do end up going the temp spinning rust route, get a couple 6TB drives for $150ea, mirror them. Don't trust a single 8TB by itself for this amount of data shuffling.

    0 Not allowed!
    771 Racing -=- FaceBook
    Backed by Heroic Racing Apparel
    A-ko - 1992 FZR400RR SP / Mistake - 1992 WR250ZD / Facade - 2003 YZF-R6 / ??? - 2009 XC-W 300

  11. #11
    Lifer Kurlon's Avatar
    Join Date
    Jun 2007
    Location
    Waterboro ME
    Age
    40
    Posts
    11,743
    iTrader Score
    0

    Re: Computer Nerds - Short term / High Volume Cloud Storage

    If you DO go the Clayton's Bits'R'US route, there are ways to cram that data over small pipes while maintaining encryption and keeping speed up.

    0 Not allowed!
    771 Racing -=- FaceBook
    Backed by Heroic Racing Apparel
    A-ko - 1992 FZR400RR SP / Mistake - 1992 WR250ZD / Facade - 2003 YZF-R6 / ??? - 2009 XC-W 300

  12. #12
    Lifer loudbeard's Avatar
    Join Date
    Jun 2013
    Location
    Rehoboth MA
    Age
    34
    Posts
    2,623
    iTrader Score
    2 (100%)

    Re: Computer Nerds - Short term / High Volume Cloud Storage

    Quote Originally Posted by Crimedog View Post
    What else would you call it? I don't think people download porn any more though, why would you when you can just stream it for free on demand?
    Last night I just so fappened to find Monster's Ball on HBO at *just* the right time.

    0 Not allowed!
    I went to MMI I know what Im doing here chief

  13. #13
    Senior Member
    Join Date
    Jan 2014
    Location
    Framingham, MA
    Posts
    523
    iTrader Score
    0

    Re: Computer Nerds - Short term / High Volume Cloud Storage

    Quote Originally Posted by Kurlon View Post
    So, first question is are you using zfs compression? If you don't have that on, flip the switch. As long as you don't go heavy handed and stick to lz4 you don't get much of a speed hit (in some cases it's actually quicker) and stuff like RAW files end up taking up much less space. You'll need to remove and replace the files to get them compressed but you can do that with much smaller storage and a little planning. I think there are scripts out there that will automate the process for you in a ZFS env.

    Check out rsync.net, and Backblaze for online dumping ground options. If you do end up going the temp spinning rust route, get a couple 6TB drives for $150ea, mirror them. Don't trust a single 8TB by itself for this amount of data shuffling.
    I do have compression on everything but the video files, iirc it was suggested to turn it off on them because they are either direct streamed or transcoded and it takes less resources. Not sure what the compression ration is but I can check when I get home. For backblaze it looks like I'd need their enterprise shit (like rsync) and any of that is charged per gb and 5.5TB x $0.06/GB = $330/mo. Some services charge for data in/out too. I don't terribly mind picking up hdds but it seems wasteful. Also I'd need more hardware because once I rebuild my ZFS pool I've used all my sata ports.

    0 Not allowed!

  14. #14
    Lifer Kurlon's Avatar
    Join Date
    Jun 2007
    Location
    Waterboro ME
    Age
    40
    Posts
    11,743
    iTrader Score
    0

    Re: Computer Nerds - Short term / High Volume Cloud Storage

    ZFS will skip compression on blocks that don't test to be compressible to avoid spinning CPU time where it's not going to help, so doing compress=on (and not forcing gzip) shouldn't have a negative impact when dealing with videos. So, yeah, you may need more than 6TB of storage to back that up depending on what you push to.

    If you're going to have to spend $330 for one month of Backblaze / rsync.net / etc, that'll cover two 6TB drives. Another $100 or $150 will get you a four bay, no raid USB / eSATA enclosure. Move the unused drives into it when done as extra ZFSable storage, go gzip-9 on compression and call it cold storage.

    0 Not allowed!
    771 Racing -=- FaceBook
    Backed by Heroic Racing Apparel
    A-ko - 1992 FZR400RR SP / Mistake - 1992 WR250ZD / Facade - 2003 YZF-R6 / ??? - 2009 XC-W 300

  15. #15
    Senior Member
    Join Date
    Jan 2014
    Location
    Framingham, MA
    Posts
    523
    iTrader Score
    0

    Re: Computer Nerds - Short term / High Volume Cloud Storage

    Quote Originally Posted by Kurlon View Post
    ZFS will skip compression on blocks that don't test to be compressible to avoid spinning CPU time where it's not going to help, so doing compress=on (and not forcing gzip) shouldn't have a negative impact when dealing with videos. So, yeah, you may need more than 6TB of storage to back that up depending on what you push to.

    If you're going to have to spend $330 for one month of Backblaze / rsync.net / etc, that'll cover two 6TB drives. Another $100 or $150 will get you a four bay, no raid USB / eSATA enclosure. Move the unused drives into it when done as extra ZFSable storage, go gzip-9 on compression and call it cold storage.
    Good to know, thanks! I'm probably being naive but a 4x3tb zfs1 pool has a usable 6tb and a 6x3tb will get me to 10tb usable and that had just better be enough or I'll start deleting shit. Or as you suggest, cold storage. Either way I was hoping that I could skate by with something like a month of google drive 10tb storage @ $100 or find some angel cloud data type.

    0 Not allowed!

  16. #16
    Lifer Kurlon's Avatar
    Join Date
    Jun 2007
    Location
    Waterboro ME
    Age
    40
    Posts
    11,743
    iTrader Score
    0

    Re: Computer Nerds - Short term / High Volume Cloud Storage

    If you do 6x2 in a mirror, you can add pairs of drives to expand instead of added in 3 drive sets. In other news, there is supposed to be a demo of live ZFS single drive raid-set expansion at an upcoming conference so that'll be cool.

    0 Not allowed!
    771 Racing -=- FaceBook
    Backed by Heroic Racing Apparel
    A-ko - 1992 FZR400RR SP / Mistake - 1992 WR250ZD / Facade - 2003 YZF-R6 / ??? - 2009 XC-W 300

  17. #17
    Lifer
    Join Date
    Nov 2009
    Location
    Bristol County
    Age
    30
    Posts
    2,516
    iTrader Score
    0

    Re: Computer Nerds - Short term / High Volume Cloud Storage

    Quote Originally Posted by Kurlon View Post
    If you DO go the Clayton's Bits'R'US route, there are ways to cram that data over small pipes while maintaining encryption and keeping speed up.
    5.5TB = 5,500,000 MB / 150mbps (18.75 M/sec) = 293,333 sec / 60 / 60 / 24 = 3.4 days. That's assuming 150mbps upload, and no performance bottlenecks from drive I/O or transfer protocol. Which seems an unlikely assumption.

    It can be done, but don't forget to consider the time required to transfer that much data to a cloud option. Half that upload speed and it's 2 weeks roundtrip assuming zero downtime.

    0 Not allowed!
    02 Aprilia Futura
    86 GPz900R

  18. #18
    Posting Freak Tekime's Avatar
    Join Date
    Mar 2016
    Location
    Falmouth, ME
    Age
    38
    Posts
    836
    iTrader Score
    0

    Re: Computer Nerds - Short term / High Volume Cloud Storage

    Office365 is about $12/mo and includes 1TB per user up to 5 users. Two subscriptions = 10TB across 10 accounts for like $25/mo.

    Can't even begin to guess on transfer speeds, juggling 10 accounts, wouldn't want to try it myself but just... thinking outside the box here.

    0 Not allowed!
    05GSXR75005SV65090DR350

  19. #19
    Lifer Kurlon's Avatar
    Join Date
    Jun 2007
    Location
    Waterboro ME
    Age
    40
    Posts
    11,743
    iTrader Score
    0

    Re: Computer Nerds - Short term / High Volume Cloud Storage

    Quote Originally Posted by aldend123 View Post
    5.5TB = 5,500,000 MB / 150mbps (18.75 M/sec) = 293,333 sec / 60 / 60 / 24 = 3.4 days. That's assuming 150mbps upload, and no performance bottlenecks from drive I/O or transfer protocol. Which seems an unlikely assumption.

    It can be done, but don't forget to consider the time required to transfer that much data to a cloud option. Half that upload speed and it's 2 weeks roundtrip assuming zero downtime.
    Unless his data is crazy fragmented, his 3tb x 4 setup should be able to sustain that throughput pretty well. His CPU is only dual core, so normally I'd toss pixz on there to compress it further but that i3 won't do it at 150Mb/s I don't think. The nice thing with using rsync.net is that it's good at incremental pushes with interruptions, and it can checksum at both ends so you can be confident that what you've pushed up has actually been pushed up. Then repeat to pull it back. So if it takes a week each dir, just let it grind.

    0 Not allowed!
    771 Racing -=- FaceBook
    Backed by Heroic Racing Apparel
    A-ko - 1992 FZR400RR SP / Mistake - 1992 WR250ZD / Facade - 2003 YZF-R6 / ??? - 2009 XC-W 300

  20. #20
    Burns retinas nhbubba's Avatar
    Join Date
    Jun 2011
    Location
    Seacoast NH
    Posts
    15,036
    iTrader Score
    1 (100%)

    Re: Computer Nerds - Short term / High Volume Cloud Storage

    That DRZ was totally worth the $..

    0 Not allowed!

  21. #21
    Super Moderator TheIglu's Avatar
    Join Date
    Apr 2003
    Location
    Royalston, MA
    Age
    37
    Posts
    18,505
    iTrader Score
    2 (100%)

    Re: Computer Nerds - Short term / High Volume Cloud Storage

    In any case, if you need some primo box to use for teh temps, provided it's encrypted data and nothing ills of the legal, I can get you some space to dabble. I have a 500Mbps synchronus pipe that will be virtually unutilized starting in a few days until January, so let me know if you want me to heat that old bitch up.

    0 Not allowed!
    -Clayton
    2006 Suzuki SV650
    2004 Suzuki Vstrom 650
    2001 Kawasaki ZRX1200R
    1986 Honda Oldwing Medicaide
    1982 Honda CB750F Super Sport

  22. #22
    Super Moderator TheIglu's Avatar
    Join Date
    Apr 2003
    Location
    Royalston, MA
    Age
    37
    Posts
    18,505
    iTrader Score
    2 (100%)

    Re: Computer Nerds - Short term / High Volume Cloud Storage

    Quote Originally Posted by nhbubba View Post
    That DRZ was totally worth the $..
    Yeah, it was, but finding a right buyer for the right machine is a waiting game. Don't take it personal, C-dog. It's just business.

    0 Not allowed!
    -Clayton
    2006 Suzuki SV650
    2004 Suzuki Vstrom 650
    2001 Kawasaki ZRX1200R
    1986 Honda Oldwing Medicaide
    1982 Honda CB750F Super Sport

  23. #23
    Super Moderator TheIglu's Avatar
    Join Date
    Apr 2003
    Location
    Royalston, MA
    Age
    37
    Posts
    18,505
    iTrader Score
    2 (100%)

    Re: Computer Nerds - Short term / High Volume Cloud Storage

    Sia may be your ticket for cheap cloud storage. Right now it's running about $0.99/TB/Month.

    0 Not allowed!
    -Clayton
    2006 Suzuki SV650
    2004 Suzuki Vstrom 650
    2001 Kawasaki ZRX1200R
    1986 Honda Oldwing Medicaide
    1982 Honda CB750F Super Sport

  24. #24
    Lifer Kurlon's Avatar
    Join Date
    Jun 2007
    Location
    Waterboro ME
    Age
    40
    Posts
    11,743
    iTrader Score
    0

    Re: Computer Nerds - Short term / High Volume Cloud Storage

    Quote Originally Posted by TheIglu View Post
    Sia may be your ticket for cheap cloud storage. Right now it's running about $0.99/TB/Month.
    Block chain all the things! (Initial start up time can take as LITTLE as a few hours, or a few DAYS just to initially validate the blockchain, EEEK!)

    0 Not allowed!
    771 Racing -=- FaceBook
    Backed by Heroic Racing Apparel
    A-ko - 1992 FZR400RR SP / Mistake - 1992 WR250ZD / Facade - 2003 YZF-R6 / ??? - 2009 XC-W 300

  25. #25
    Super Moderator TheIglu's Avatar
    Join Date
    Apr 2003
    Location
    Royalston, MA
    Age
    37
    Posts
    18,505
    iTrader Score
    2 (100%)

    Re: Computer Nerds - Short term / High Volume Cloud Storage

    Only took me a couple hours initially, but I have a serious desktop CPU that was crunchin away.

    0 Not allowed!
    -Clayton
    2006 Suzuki SV650
    2004 Suzuki Vstrom 650
    2001 Kawasaki ZRX1200R
    1986 Honda Oldwing Medicaide
    1982 Honda CB750F Super Sport

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •