Testing Chat IV: A New Hope

[adr]

OMG THE FILE TRANSFER IS FINALLY COMPLETE

tehre's still other project files but now I can just install sql server and take this shit live and the client's long downtime can be ended

FYI apparently transferring one 3 GB file takes 15 minutes where transferring a total of 6 GB but spread across 170,000 files takes 3 hours

[Oxymoron]

This does not surprise me.

[evilsoup]

isn't this why zip/tar exists

[adr]

I thought of zipping it but when it estimated 2 hours to do that (and these estimations are almost always underestimates) and the fact that the two boxes are currently sharing a local area connection I figured the bandwidth should be fat enough that it'd be done before it finished zipping anyway

[evilsoup]

you can tar and stream over a network on the fly (packing it up on one side and unpacking it on the other), you can probably do the same with zip
don't know if that would have actually helped in your case or not

[Oxymoron]

Tarred packets ? Is that the same thing as the option people have to tar elements of their web pages and have the browser un-tar them upon reception, or is it something else ?

[evilsoup]

I don't really know anything about websites beyond HTML and a little javascript, so... maybe?

tar is a *nix file archive thing. If you have loads of files you need to copy, many other tools will spawn a separate process for each file, which might be the cause of adr's problem; whereas tar doesn't do this. You can pipe the STDOUT of the tar through the network to another computer (with ssh or w/e), and then unpack it on the other side, with only the one tar process (well, two -- one on each computer).

I don't know if zip has the same capability, or if there's another windows tool that does the same thing

EDIT: here's a link, or just google 'tar over network'.

[adr]

i just dragged and dropped the relevant folder between the two computers (they are running windows server 2008). idk what happened behind the scenes

but my guess is the main thing has to do with hard drive seeking. pulling individual files means getting their metadata then seeking to the file (which itself may be fragmented), loading it up, sending it, and on the other side, writing it back out

then repeat for everything.


with one large file it is much more sequential, less jumping around the disk to get all the files. (i'm pretty sure Windows does some work to try to minimize this - before it did the copy, it spent several minutes preparing to do the copy, which is surely gathering a list of stops and lining them up. but still it has got to be dancing across the drive)

[Oxymoron]

I can't wait for SSDs to mature enough to become competitive with hard disk drives.

[adr]

Is that the same thing as the option people have to tar elements of their web pages and have the browser un-tar them upon reception, or is it something else ?

you might be thinking of gzip

you can gzip websites and the browser unzips them

you can also gzip tar files, which is common on unix land, hence the extension you can see .tar.gz or .tgz, meaning gzipped tar

you can also gzip stuff sequentially so like

tar c stuff | gzip | network-app

and on the other side

network-app | gunzip | tar x

would work to create a stream of stuffs

[evilsoup]

yeah, hard drive seeking is probably the real bottleneck

[adr]

so a mistake i've made like ten times now but never learn from

i set up company.com, but not www.company.com

takes liek 5 seconds to add it but i never think to do it until someone else uses the www variant and it doesn't work

[xon]

I can't wait for SSDs to mature enough to become competitive with hard disk drives.

They already are. What needs to happen is for people to stop saying "omg I need the slowest 1tb spinning rust on the market" when they can get a 256gb SSD for the same price.

Sure, they aren't competitive for bulk storage but for OS drives and VMs? They blow everything else out of the water cost wise for the performance.

[zhaktronz]

I can't wait for SSDs to mature enough to become competitive with hard disk drives.

They already are. What needs to happen is for people to stop saying "omg I need the slowest 1tb spinning rust on the market" when they can get a 256gb SSD for the same price.

Sure, they aren't competitive for bulk storage but for OS drives and VMs? They blow everything else out of the water cost wise for the performance.

Or even get both; 128gb for os and crappy cheap mechanical for bulk data

[Oxymoron]

Yeah, I know that. But what I meant is : I can't wait for SSDs to become as big and cheap-per-GB as the good old spinners.

Coupled with the next generation of SATA and 64-bit PC architectures fully supporting the Many Cores CPU architectures and with 16-32 GB of RAM in standard... It's going to be so smooth !

I'd say that by 2020-2025 we should see that thing becoming standard for consumer grade computers.

[weemadando]

That's my latest system. SSD for OS, programs and maybe some games eventually. 1TB drive for games. 3TB drive for bulk storage of everything.

[Phantasee]

My SSD is full. 128 GB drive. Trying to figure out what the hell I stored on it. My 1TB storage drive is nearly empty, I'm not actually sure I have more on it than I do on the SSD.

[joviwan]

your flagrantly irresponsible data management makes my eyebrow twitch

[phongn]

I can't wait for SSDs to mature enough to become competitive with hard disk drives.

They already are. What needs to happen is for people to stop saying "omg I need the slowest 1tb spinning rust on the market" when they can get a 256gb SSD for the same price.

Spinning glass, sir.

Sure, they aren't competitive for bulk storage but for OS drives and VMs? They blow everything else out of the water cost wise for the performance.

Memristors appear to be the holy grail for mass high speed storage and scale way better than flash (which is hitting seriously limits, though there are tricks like thermal reannealing that might extend it(.

[RogueIce]

So if I had $7,000 with nothing better to spend it on, I could totally buy a kickass Alienware gaming PC.

But I do not have $7,000 with nothing better to spend it on.

[Darksi4190]

Man. I can't believe some of the shit they were doing in Cartoons back in the 90s.

[Flagg]

I miss Speedy Gonzalez. And his incredibly racist cousin Slowpoke Rodriguez.

[adr]

so with the government's insistence that metadata doesn't have a reasonable expectation of privacy

maybe we should just publicize all cia logs. not the content of the conversations, just metadata of who talked to whom, when, and where both parties were

LOL

[evilsoup]

government secrecy is anti-democratic
the NSA etc. are enemies of democracy, albeit hopefully well-meaning ones (but I have no way of knowing for sure, by definition)
corporate secrecy is also potentially harmful
personal secrecy/privacy should be protected, however

hmm... perhaps an analogy could be made with personal vs. private property

[uraniun235]

Yeah, I know that. But what I meant is : I can't wait for SSDs to become as big and cheap-per-GB as the good old spinners.

I don't see this as likely any time soon. If we converted every single chip fab in the world to making solid state storage tomorrow, it still wouldn't be sufficient to meet the bulk storage demand currently being filled by platter drive sales and production.