#ceph IRC Log


IRC Log for 2011-09-23

Timestamps are in GMT/BST.

[0:00] * aliguori (~anthony@ Quit (Remote host closed the connection)
[0:04] * pruby (~tim@leibniz.catalyst.net.nz) Quit (Ping timeout: 480 seconds)
[0:06] * pruby (~tim@leibniz.catalyst.net.nz) has joined #ceph
[0:15] * pruby (~tim@leibniz.catalyst.net.nz) Quit (Ping timeout: 480 seconds)
[0:16] * pruby (~tim@leibniz.catalyst.net.nz) has joined #ceph
[0:34] * aliguori (~anthony@cpe-70-123-132-139.austin.res.rr.com) has joined #ceph
[0:52] <sagewk> tv|work: is there a script to rebuild man/* from doc/man/*?
[0:55] <joshd> I think it was documented in the commit message
[0:57] <sagewk> ah i see it
[0:58] * cp (~cp@ip-64-134-24-179.public.wayport.net) has joined #ceph
[0:59] <Tv|work> sagewk: yeah we call that "cp"
[0:59] <Tv|work> ;)
[0:59] <sagewk> and admin/build-doc :)
[0:59] <Tv|work> yeah
[1:00] <Tv|work> i also have a local admin/diff-man that i didn't commit, that may be interesting if you're actively working on the manpages
[1:00] <Tv|work> diffs the formatted *roffs
[1:01] <Tv|work> but it assumes meld etc, i didn't want to push my tools too hard
[1:02] <sagewk> i was going to add a rule to one of the makefiles to build-doc and copy them over
[1:03] <sagewk> anyone care if i merge wip-names sooner rather than later?
[1:04] <sjustlaptop> sooner is probably better
[1:05] <Tv|work> sagewk: i didn't put build-doc in makefiles as it has pretty heavy build-deps, downloads stuff from the net & runs it, etc
[1:05] <sagewk> i wouldn't include it in make all.. you'd need to run it explicitly
[1:06] <Tv|work> sagewk: yeah my fingers do ./ad TAB bu TAB faster than "make " ...umm what was the name of the target... etc
[1:06] <Tv|work> totally personal preference there ;)
[1:09] <Tv|work> and there's almost no incrementalism there, so make can't do smart things
[1:10] <sagewk> sigh.. well teh hard part is making build-doc run in the first place
[1:16] <sagewk> merged!
[1:16] <Tv|work> (or, i should say, sphinx does incremental inside itself)
[1:19] <sjustlaptop> sagewk: what kernel version is benjamin being updated to?
[1:19] <sjustlaptop> rc4 does show problems when doing repeated watch-notify + thrashing, but rc7 seems to have taken care of it
[1:20] <sagewk> sjustlaptop: interesting!
[1:21] <sjustlaptop> so far, anyway
[1:21] <sagewk> the kernel i just booted hit a BUG_ON on every cosd process in the cluster, so I'm about to do a new one.
[1:21] <sagewk> did you see anything suspicious in the btrfs shortlog during that period that might explain the bug/fix?
[1:34] * cp (~cp@ip-64-134-24-179.public.wayport.net) Quit (Quit: cp)
[1:39] * adjohn (~adjohn@ Quit (Quit: adjohn)
[2:01] * votz_ is now known as votz
[2:03] * greglap (~Adium@aon.hq.newdream.net) Quit (Quit: Leaving.)
[2:08] * sjustlaptop (~sam@aon.hq.newdream.net) Quit (Quit: Leaving.)
[3:02] * Tv|work (~Tv|work@aon.hq.newdream.net) Quit (Ping timeout: 480 seconds)
[3:02] * adjohn (~adjohn@50-0-92-177.dsl.dynamic.sonic.net) has joined #ceph
[3:10] * sjustlaptop (~sam@96-41-121-194.dhcp.mtpk.ca.charter.com) has joined #ceph
[3:11] * sjustlaptop (~sam@96-41-121-194.dhcp.mtpk.ca.charter.com) Quit ()
[3:14] * bchrisman (~Adium@70-35-37-146.static.wiline.com) Quit (Quit: Leaving.)
[3:30] * jojy (~jojyvargh@70-35-37-146.static.wiline.com) Quit (Quit: jojy)
[3:31] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[3:36] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[4:35] * adjohn (~adjohn@50-0-92-177.dsl.dynamic.sonic.net) Quit (Quit: adjohn)
[4:36] * jojy (~jojyvargh@75-54-231-2.lightspeed.sntcca.sbcglobal.net) has joined #ceph
[4:36] * jojy (~jojyvargh@75-54-231-2.lightspeed.sntcca.sbcglobal.net) Quit ()
[4:37] * jojy (~jojyvargh@75-54-231-2.lightspeed.sntcca.sbcglobal.net) has joined #ceph
[4:37] * jojy (~jojyvargh@75-54-231-2.lightspeed.sntcca.sbcglobal.net) Quit ()
[4:47] * greglap (~Adium@mobile-166-205-142-189.mycingular.net) has joined #ceph
[5:22] * greglap (~Adium@mobile-166-205-142-189.mycingular.net) Quit (Quit: Leaving.)
[5:33] * Nightdog (~karl@190.84-48-62.nextgentel.com) Quit (Ping timeout: 480 seconds)
[5:35] * Nadir_Seen_Fire (~dantman@S0106001731dfdb56.vs.shawcable.net) has joined #ceph
[5:40] <ajm> is there any negative consequence to referencing non-existent devices in your crush map?
[5:41] * DanielFriesen (~dantman@S0106001731dfdb56.vs.shawcable.net) Quit (Read error: Operation timed out)
[6:42] * bchrisman (~Adium@c-98-207-207-62.hsd1.ca.comcast.net) has joined #ceph
[6:46] <sage> ajm: if crush has a weight on them, it'll make things a bit less efficient, because it'll have to retry placement when it picks them
[6:48] <ajm> thanks sage
[6:48] <sage> np
[7:28] * adjohn (~adjohn@50-0-92-177.dsl.dynamic.sonic.net) has joined #ceph
[8:41] * cp (~cp@c-98-234-218-251.hsd1.ca.comcast.net) has joined #ceph
[8:53] * cp (~cp@c-98-234-218-251.hsd1.ca.comcast.net) Quit (Quit: cp)
[9:09] * adjohn (~adjohn@50-0-92-177.dsl.dynamic.sonic.net) Quit (Quit: adjohn)
[9:15] * Nadir_Seen_Fire (~dantman@S0106001731dfdb56.vs.shawcable.net) Quit (Remote host closed the connection)
[9:21] * Dantman (~dantman@S0106001731dfdb56.vs.shawcable.net) has joined #ceph
[12:57] * The_Bishop (~bishop@sama32.de) has joined #ceph
[14:11] <lxo> sjust, I ended up building the patched cosd and it successfully completed the conversion
[16:16] <wonko_be> all, is there some info available on the amount of ram needed when using a rbd device?
[16:16] <wonko_be> a test machine went out-of-memory just doing a rsync to a rbd device
[16:17] <wonko_be> (xfs on the rbd, data came from another machine)
[16:48] * jmlowe (~Adium@140-182-215-74.dhcp-bl.indiana.edu) has joined #ceph
[17:01] * adjohn (~adjohn@50-0-92-177.dsl.dynamic.sonic.net) has joined #ceph
[18:30] <gregaf1> wonko_be: hmm, that's not good — what version?
[18:37] * kloo (~kloo@a82-92-246-211.adsl.xs4all.nl) has joined #ceph
[18:38] <kloo> hello.
[18:38] <gregaf1> hello
[18:39] <kloo> i've played with ceph some time ago.
[18:39] <kloo> now i'd like to start using it more seriously but there's one thing i'm not clear on.
[18:40] <kloo> how (well) does ceph handle osds of different sizes?
[18:40] <sjust> we did add some osd balancing stuff within the last year
[18:40] <gregaf1> sjust: we've had osd balancing stuff for a while, I thought
[18:41] <gregaf1> you can set weights on each OSD proportional to their disk size, and the system will place data appropriately
[18:41] <kloo> ah ok.
[18:41] <gregaf1> having more data on some nodes than others will unbalance the IO patterns, but everything will work fine, just potentially a bit slower than peak
[18:41] <kloo> what happens when an osd runs out of space?
[18:42] <gregaf1> everything stops
[18:42] <gregaf1> it's not good; you shouldn't let that happen
[18:42] <gregaf1> but you can just pop in a new OSD and data will rebalance and you can get on with your life
[18:43] <kloo> ok. :)
[18:43] <kloo> pg splitting is still work in progress, right?
[18:44] <gregaf1> yeah, sjust knows more about the details of that
[18:44] <sjust> kloo: yeah, much of the groundwork has been laid, but it's not yet done
[18:44] <sjust> the on disk format changes in 0.35 are related
[18:44] <kloo> when i was previously looking into ceph, sage told me i should have enough pgs for expected growth, from the start.
[18:45] <kloo> .. or enough pgs to keep going until pg splitting works. :)
[18:45] <kloo> my thanks to both of you.
[19:00] * Tv|work (~Tv|work@aon.hq.newdream.net) has joined #ceph
[19:01] * jojy (~jojyvargh@70-35-37-146.static.wiline.com) has joined #ceph
[19:01] * Dantman (~dantman@S0106001731dfdb56.vs.shawcable.net) Quit (Read error: Connection reset by peer)
[19:34] * kloo (~kloo@a82-92-246-211.adsl.xs4all.nl) Quit (Ping timeout: 480 seconds)
[19:42] * tjikkun (~tjikkun@2001:7b8:356:0:225:22ff:fed2:9f1f) Quit (Quit: Ex-Chat)
[19:53] * greglap (~Adium@aon.hq.newdream.net) has joined #ceph
[19:55] * cp (~cp@c-98-234-218-251.hsd1.ca.comcast.net) has joined #ceph
[20:12] * jmlowe (~Adium@140-182-215-74.dhcp-bl.indiana.edu) Quit (Quit: Leaving.)
[20:20] * Dantman (~dantman@S0106001731dfdb56.vs.shawcable.net) has joined #ceph
[20:28] * greglap (~Adium@aon.hq.newdream.net) Quit (Quit: Leaving.)
[21:10] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[21:16] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[21:23] * jmlowe (~Adium@ has joined #ceph
[21:33] <lxo> sjust, is it normal that, even after the osd finished the conversion and ran for a while, upon the next restart it prints one of those “Moving corrupted log file” for each PG *again*, taking a very long time to go through them all? shouldn't it go straight to the new logs and be done with it?
[21:35] <jojy> i am getting build error on the latest pull from HEAD
[21:36] <jojy> perfglue/heap_profiler.cc:47: error: ‘IsHeapProfilerRunning’ was not declared in this scope
[21:36] * bchrisman (~Adium@c-98-207-207-62.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[21:53] <Tv|work> jojy: you'll need to ./autogen.sh etc due to commit 07bf9b821c802501056528236ff034f68bec1e3f, i think
[21:53] * jmlowe (~Adium@ Quit (Quit: Leaving.)
[21:55] <gregaf1> that's…a pretty old commit
[21:56] <Tv|work> oh err
[21:56] <Tv|work> i just hunted for mentions of IsHeap...
[21:56] <gregaf1> not that there's anything very new in the perfglue code, it was last touched Aug 17 and that didn't do anything
[21:56] <Tv|work> yeah how would that have come up now
[21:58] <Tv|work> jojy: ohh that might be caused by the source filename rewrite c* -> ceph-*
[22:00] <jojy> i wonder why its not showing up for u guys
[22:00] <Tv|work> well, the gitbuilder runs git clean -fdx between builds, etc
[22:03] * bchrisman (~Adium@ has joined #ceph
[22:04] * cp (~cp@c-98-234-218-251.hsd1.ca.comcast.net) Quit (Quit: cp)
[22:05] <jojy> ic
[22:10] <jojy> where is the def for "IsHeapProfilerRunning" ?
[22:10] <Tv|work> i'm running an incremental build over those changes without any autogen.sh etc runs, we'll see how that goes
[22:10] <Tv|work> jojy: in the google perftools lib i think
[22:11] <jojy> does configuring with --without-tcmalloc has anything to do
[22:17] * sjustlaptop (~sam@aon.hq.newdream.net) has joined #ceph
[22:17] <Tv|work> sagewk: hey i still see cmon(8) etc in manpages -- mind if i fix, or are you still doing that work?
[22:17] <Tv|work> (i have other changes rolling in)
[22:19] <jojy> Tvjwork: it was "--without-tcmalloc"
[22:19] <Tv|work> jojy: sounds like a bug!
[22:19] <jojy> i had to reconfigure without tcmalloc
[22:20] <gregaf1> jojy: hmmm, if you're referencing heap_profiler.cc and you're building without tcmalloc then something's busted
[22:20] <gregaf1> maybe the detection is done wrong; which commit were you on previously that built correctly?
[22:20] <jojy> gregaf1: no when i configure "without tcmalloc" it build fine
[22:21] <jojy> sorry for confusing u
[22:21] <jojy> :|
[22:21] <gregaf1> hrm
[22:21] <gregaf1> you had it building correctly on an earlier version, right?
[22:39] <sagewk> tv|work: go ahead and fix
[22:39] <Tv|work> sagewk: i also see some old names in spec etc, will fix those too
[22:40] <sagewk> k
[23:34] <joshd> sagewk: wip-names merge broke the debian gitbuilder as well
[23:39] <wonko_be> gregaf1: cluster is .35, the client is a stock debian 2.6.39 kernel (wheezy i think)
[23:39] <gregaf1> wonko_be: and the client is the one running out of memory?
[23:39] <gregaf1> or one of the OSDs
[23:40] <wonko_be> the client
[23:40] <wonko_be> where we did the "echo ... > ... add"
[23:40] <wonko_be> and the mkfs.xfs, the mount and the rsync
[23:40] <wonko_be> the cluster is fine
[23:40] <gregaf1> what implementation of rbd are you using? qemu, kernel, ...
[23:41] <wonko_be> hmmm, good question... just the kernel one I assume
[23:41] <wonko_be> modprobe rbd and then ... > /sys/.../add
[23:41] <wonko_be> didn't know there were more than one
[23:41] <gregaf1> okay
[23:42] <wonko_be> we create the rbd image on the cluster, add it on the client, format it, and then sync 1 TB of data to it using rsync
[23:42] <gregaf1> I wonder if you've got it set up so that it was dirtying data faster than it could flush it out
[23:42] <gregaf1> I would think the kernel would prevent that from happening, but it might be possible
[23:42] <gregaf1> joshd: you have any ideas?
[23:43] <wonko_be> it is all connected in one 1gbps network
[23:43] <wonko_be> 3 ceph nodes, 1 client, 1 machine-where-we-rsync-data-from
[23:43] <wonko_be> if you want, i can try to trigger it again, but you'll have to give me some pointers what I should setup to debug
[23:44] <joshd> I don't think the kernel should let that happen, but I'm not the most familiar with debugging it
[23:44] <gregaf1> yeah, me neither :
[23:44] <gregaf1> :/
[23:45] <wonko_be> a sidenote, the client is a xen dom0
[23:45] <wonko_be> but it is NOT inside a domU we are doing the sync
[23:45] <wonko_be> just on the dom0
[23:45] <joshd> also, with .39, I think the queue block size is small enough that it'd take tons of in-flight requests to use all that memory
[23:45] <wonko_be> as for bandwidth
[23:46] <gregaf1> joshd: what I'm wondering is if the page cache is getting dirtied super-fast due to the 1TB rsync, but it can't get flushed out very quickly
[23:46] <wonko_be> the client and the cluster is on a single switch, 1gpbs, the client to sync from is further away only 100mbsp
[23:46] <wonko_be> mbps that is
[23:46] <gregaf1> because in .39 it would be doing sync writes, wouldn't it?
[23:46] <gregaf1> and actually that might be the one that was doing very small writes, too?
[23:46] <joshd> it's still doing sync writes, in .39 it's just doing smaller ones
[23:47] <gregaf1> yeah, so I wonder if the kernel memory handling isn't dealing with that properly
[23:47] <gregaf1> it's the only thing I can think of, but it just seems so unlikely
[23:47] <wonko_be> any way I can debug this?
[23:47] <wonko_be> or help any way?
[23:47] <joshd> it's sounds likely, unless our queue size is unbounded or something silly like that
[23:47] <gregaf1> wonko_be: our proper kernel devs are busy elsewhere right now, but they'll have some ideas when they get back :)
[23:48] <wonko_be> I'll keep the "bad" client as it is
[23:48] <wonko_be> I can restart the rsync any time
[23:48] <wonko_be> no idea if it will go OOM again, but i'm certainly willing to help out
[23:52] * cp (~cp@adsl-76-194-112-188.dsl.pltn13.sbcglobal.net) has joined #ceph
[23:52] <wonko_be> well, I'm around, and I'll keep the current situation like it is, so just give a shout

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.