#ceph IRC Log


IRC Log for 2011-03-11

Timestamps are in GMT/BST.

[21:53] -solenoid.oftc.net- *** Looking up your hostname...
[21:53] -solenoid.oftc.net- *** Checking Ident
[21:53] -solenoid.oftc.net- *** No Ident response
[21:53] -solenoid.oftc.net- *** Found your hostname
[21:53] * CephLogBot (~PircBot@fubar.widodh.nl) has joined #ceph
[21:54] <wido> crappy Java bot sometimes quits...
[21:54] <cmccabe> and as we've already discussed, you seldom even get near 125 MB/s with gigabit, whereas SATA can get pretty close to it
[21:54] <cmccabe> so they are bandwidth limited
[21:54] <johnl> there is ethernet bonding
[21:54] <gregaf> cmccabe: as we said, essentially all their data input is coming from people uploading on consumer internet lines
[21:54] <cmccabe> so I imagine most of those drives spend a lot of time spun down
[21:55] <gregaf> so 2mb/sec at the outside
[21:55] <gregaf> and yes, plenty of spin-down time
[21:55] <gregaf> probably the vast majority of the drives in their DC are idle most of the time
[21:55] <darkfader> and also no appends, so they can just fill one drive after another
[21:56] <gregaf> wido: you could bond the drives together with btrfs in one filesystem (or a RAID card), although you'd have one hell of a bandwidth wall
[21:56] <cmccabe> johnl: only one NIC on this, so no bonding
[21:56] <cmccabe> they don't really need it, it would seem.
[21:57] <wido> gregaf: Yes, getting that 48TB out of the "door" would be a nightmare
[21:57] <johnl> high availability would be a good enough reason, heh
[21:57] <wido> not to think one of those machines failing, would result in a lot of data movement
[21:59] <wido> I'll stick to the 4x2TB per machine with the nice Atom :)
[21:59] <johnl> they have 45 drives per device wrapped in 3 raid6 volumes
[22:00] <johnl> using JFS
[22:00] <johnl> and they have some https interface to the data
[22:00] <darkfader> JFS?
[22:00] <johnl> so they're encrypting all data in and out of those things too, even over their local systems
[22:01] <darkfader> that's the thing no AIX user has touched since 2003
[22:01] <darkfader> that really does it for me
[22:01] <johnl> http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/
[22:01] <wido> Yeah, it's pretty nice what they've done
[22:01] * darkfader runs
[22:04] <cmccabe> it would be interesting to learn why they chose JFS
[22:04] <cmccabe> kind of an offbeat choice it would seem
[22:04] <johnl> yeah, am wondering that myself atm
[22:04] <johnl> suppose they know their access patterns
[22:04] <johnl> and benchmarked
[22:05] <cmccabe> well, it has extents
[22:05] <johnl> yeah, it's pretty good fs
[22:05] <johnl> wikipedia article on it is pretty in depth
[22:05] <darkfader> johnl: it's really completely outdated
[22:05] <Tv> also consider as you're bitching about IO of the backblaze hardware: this might be a "hierarchical storage" thing, where these machines host the stable data
[22:05] <johnl> outdated?
[22:06] <darkfader> ibm gave the crumbles to us linux users
[22:06] <johnl> it's journalled and extent based
[22:06] <johnl> that's pretty modern :)
[22:06] <gregaf> I thought JFS was the most seriously log-based FS in modern use — maybe I'm mistaken about that though
[22:06] <darkfader> in aix 5 it there was the successor
[22:06] <darkfader> and by now it's long deprected
[22:06] <darkfader> so we're talking 2003/2004
[22:07] <cmccabe> it's kind of interesting that they're using a journalling FS at all
[22:07] <johnl> when it comes to filesystems, old is often a good thing!
[22:07] <cmccabe> google just nukes the FS from orbit when it seems bad
[22:07] <johnl> lots of people have found those bugs for you already :)
[22:07] <darkfader> johnl: then the aix users wouldnt have all dropped it
[22:07] <cmccabe> I guess when you're doing raid across so many drives you can't do that
[22:08] <darkfader> i wish i could get you some numbers, about when the one was introduced and when the next one
[22:08] <cmccabe> I have this picture of AIX as being tied to support contracts with IBM
[22:08] <cmccabe> IBM probably just upgrades people when it feels like it
[22:09] <johnl> never met an aix user, heh
[22:09] <johnl> I assume they do what the fuck ibm tell them
[22:09] <johnl> not like they have other options. can't maintain it themselves
[22:09] <darkfader> cmccabe: i just read the wikipedia article... it seems linux "jfs" is in fact mostly what is called jfs2 to the rest
[22:09] <cmccabe> well that's kind of the point of AIX I would imagine
[22:09] <darkfader> without some features
[22:09] <johnl> heh
[22:09] <cmccabe> if you wanted to admin it yourself you'd be using linux or something
[22:10] <johnl> yer.
[22:10] <darkfader> let me nitpick... you might run aix if your system matters
[22:10] <darkfader> like, when amazon or such is "not really important" in comparism
[22:10] <johnl> but I mean, that no aix user uses jfs is not good reasoning that jfs is not good :)
[22:10] <darkfader> they point was "not using it any more"
[22:10] <bchrisman> wonder if they can fill those boxes up within the lifetime of the universe.. :)
[22:10] <johnl> you run aix when you need to someone rich to blame when something goes wrong ;)
[22:11] <darkfader> johnl: hehe
[22:11] <johnl> someone richer than redhat ;)
[22:11] <yehuda_wk> Tv: changes looks fine to me
[22:11] <cmccabe> from what I hear a lot of companies just want their legacy systems to keep running
[22:11] <Tv> yehuda_wk: thx, merging
[22:12] <johnl> cmc: funny requirement that :)
[22:12] <cmccabe> so if that means staying on AIX then they stay on AIX
[22:12] <darkfader> cmccabe: it's like that most of the times yeah
[22:12] <cmccabe> because converting to something newer, while probably cheaper and better, involves risk and a large one-time charge
[22:12] <darkfader> but i've spent last 2 years moving from "real unix" to linux
[22:12] <cmccabe> darkfader: what distro did you move to?
[22:12] <darkfader> you can't imagine how many more issues, more failures, less performace, etc. we saw
[22:13] <darkfader> hp-sux to redhat
[22:13] <darkfader> redhat's outdatedness was extra fun
[22:13] <johnl> I use veritas filesystem.
[22:13] <johnl> forget this jfs nonsense
[22:13] <darkfader> hehe
[22:13] <cmccabe> haha
[22:13] <darkfader> we had vxfs with cluster volume manager and cio and all that
[22:14] <darkfader> but the design was done by other guys
[22:14] <darkfader> they manage to mess up everything
[22:14] <bchrisman> actually not so bad.. one node config'd like that backblaze would take 6.3d to fill?
[22:14] <darkfader> bchrisman: someone just come up with 5.xd
[22:14] <darkfader> but definitely long
[22:15] * panitaliemom (5138106c@ircip1.mibbit.com) has joined #ceph
[22:16] * panitaliemom (5138106c@ircip1.mibbit.com) Quit ()
[22:17] * Juul (~Juul@static.88-198-13-205.clients.your-server.de) Quit (Quit: Leaving)
[22:21] <darkfader> ah and i found the numbers jfs 1990, jfs2 2001. and linux jfs is really largely the jfs2 i know of. so no reason to run from it
[22:38] <sagewk> tv: can we give the f13 vm more cpus?
[22:38] <Tv> sagewk: sure, just power it off and clickety-click ;)
[22:39] <sagewk> it has 16 now
[22:40] <cmccabe> cool
[22:40] <cmccabe> er, how do I get in
[22:46] <Tv> sagewk, cmccabe: warning, colin is talking about gitbuilder-i386, not f13
[22:47] <cmccabe> I was more talking about fedora core 13
[22:47] <Tv> cmccabe: not with that ip address you ain't
[22:48] <cmccabe> well, I have things to do on either one
[22:48] <Tv> the f13 is behind nat only, use virt-manager
[22:48] <cmccabe> I don't understand the IP address thing. I can ssh from metropolis if that helps
[22:48] <Tv> what about the "IP address thing"?
[22:49] <sagewk> no ssh from the host even?
[22:49] <Tv> sagewk: that's doable, but it might bounce between addresses, due to dhcp
[22:49] <Tv> i don't know if the f13 runs an ssh server even
[22:50] <cmccabe> service ssh start
[22:50] <Tv> i can set it up with an ip address etc, that was just gonna happen in the 15 mins i spent on it
[22:50] <Tv> *not gonna happen
[22:50] * cmccabe (~cmccabe@ has left #ceph
[22:50] * cmccabe (~cmccabe@ has joined #ceph
[22:51] <cmccabe> chkconfig --level 2345 ssh on
[22:54] * neurodrone (~neurodron@dhcp215-189.wireless.buffalo.edu) has joined #ceph
[23:12] <Tv> sagewk: fyi configure.ac cleanup fully done now
[23:14] <sagewk> cool
[23:14] <sagewk> can help get f13 reachable one way or another?
[23:15] <sagewk> root@ceph-kvm1:~# ssh
[23:15] <sagewk> ssh: connect to host port 22: No route to host
[23:16] <cmccabe> gitbuilder-i386 may not be usable for triggering bug 876
[23:16] <cmccabe> because it has so little disk space that I can't create a >2GB rbd image
[23:17] <cmccabe> we really should be a little more generous with the disk space on these things
[23:17] <cmccabe> I mean ceph itself takes a little more than a gig when you compile it
[23:20] <sagewk> cmccabe: you can nfs mount cephbooter or flab, like the sepia* nodes do.
[23:21] <Tv> uhh guys, please don't work too much on gitbuilders :(
[23:21] <Tv> the more you touch them, the less i trust them
[23:21] <Tv> (to be pristine builders, that is)
[23:21] <Tv> and the disks are typically 20GB because it takes ages to clone one, even at that small a size
[23:22] <cmccabe> when do we clone gitbuilders
[23:22] <Tv> cmccabe: they are the result not the source
[23:22] <cmccabe> result of what?
[23:22] <Tv> not sure if gitbuilders got more disk too
[23:23] <Tv> but in general, when i need a new vm, i clone gold
[23:23] <cmccabe> do you clone using cp or something more sophisticated like qcow
[23:23] <Tv> libvirt
[23:25] <cmccabe> why don't we create a 386 VM for general use then
[23:25] <cmccabe> if it's just as simple as hitting clone
[23:25] <cmccabe> I'd rather not work on a live system, especially one that seems to constantly go down because of disk space issues
[23:26] <Tv> i actually don't have a gold i386 image, but it takes ~20min to create from scratch
[23:26] <Tv> let me do that, then
[23:26] <cmccabe> thanks
[23:27] <cmccabe> one thing that I've done in the past is had some other block device that I mount to do stuff in that's not the root, to use for building ceph and so forth
[23:27] <cmccabe> in a vm
[23:27] <Tv> yeah just can't do that in the gold image, libvirt is too dumb
[23:28] <cmccabe> that way I can copy the small root partition quickly, but not have to worry about builds filling up the small root partition
[23:37] * allsystemsarego (~allsystem@ Quit (Quit: Leaving)
[23:37] * neurodrone (~neurodron@dhcp215-189.wireless.buffalo.edu) Quit (Quit: neurodrone)
[23:39] <cmccabe> http://tracker.newdream.net/issues/880
[23:40] <Tv> cmccabe: did i just break that? i thought i tried all combos
[23:40] <cmccabe> it's not working with --with-gtk2=no or --without-gtk
[23:41] <cmccabe> --without-gtk2
[23:41] <Tv> cmccabe: and you tested bccffecb4a6be4ac34ac3c9bdd1a903388b0074f or later?
[23:41] <cmccabe> head of line
[23:41] <cmccabe> it's trying to do something with a variable named check, but I don't know what that is
[23:41] <Tv> oh i see the bug
[23:41] <Tv> will fix, should be easy
[23:42] <Tv> AS_IF([test "with_gtk2" != "xno"],
[23:42] <Tv> simple typo, bleh
[23:42] <Tv> but apparently i edited it after testing, the cardinal sin
[23:47] <Tv> fixed
[23:48] <cmccabe> still not working for me
[23:48] <Tv> what command
[23:48] <Tv> did you run autogen?
[23:48] <cmccabe> --without-gtk2
[23:48] <cmccabe> just remove your copy of gtk2 and try it
[23:48] <cmccabe> or rather your gtk2mm
[23:49] <cmccabe> removing gtkmm-devel isn't that big a deal
[23:49] <cmccabe> the non-devel might be
[23:49] <Tv> trust me i've run plenty of the variations now
[23:50] <Tv> did you actually pull 289318777c912f6c1bf057309e281472da4a3517 and run autogen.sh?
[23:51] <cmccabe> I didn't have 2893 a minute ago
[23:51] <cmccabe> but I re-pulled and it appeared
[23:52] <cmccabe> ok, that worked
[23:52] <cmccabe> honestly I can't believe the amount of boilerplate required to do a single if statement properly in automake
[23:53] <cmccabe> even Java isn't anywhere near this bad
[23:57] <Tv> it performs dumb macro expansions to generate age-old shell scripts (no functions!).. i'm surprised it works at all
[23:58] <cmccabe> the whole saw about "pigs being able to fly with sufficient thrust" comes to mind

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.