#ceph IRC Log

Index

IRC Log for 2012-11-19

Timestamps are in GMT/BST.

[0:23] * Leseb (~Leseb@5ED17881.cm-7-2b.dynamic.ziggo.nl) Quit (Quit: Leseb)
[0:28] * tryggvil (~tryggvil@17-80-126-149.ftth.simafelagid.is) Quit (Quit: tryggvil)
[0:36] * tnt (~tnt@140.20-67-87.adsl-dyn.isp.belgacom.be) Quit (Ping timeout: 480 seconds)
[0:49] * glowell (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) has joined #ceph
[1:04] * miroslav (~miroslav@173-228-38-131.dsl.dynamic.sonic.net) has left #ceph
[1:11] * Qu310 (Q@qten.qnet.net.au) has joined #ceph
[1:11] * Qten (Q@qten.qnet.net.au) Quit (Read error: Connection reset by peer)
[1:23] * Steki (~steki@bojanka.net) has joined #ceph
[1:25] * maxiz_ (~pfliu@111.194.202.136) Quit (Quit: Ex-Chat)
[1:26] * BManojlovic (~steki@137-166-222-85.adsl.verat.net) Quit (Ping timeout: 480 seconds)
[1:36] * yanzheng (~zhyan@jfdmzpr03-ext.jf.intel.com) has joined #ceph
[1:46] * sjustlaptop (~sam@68-119-138-53.dhcp.ahvl.nc.charter.com) Quit (Ping timeout: 480 seconds)
[1:52] * yoshi (~yoshi@p11251-ipngn4301marunouchi.tokyo.ocn.ne.jp) has joined #ceph
[1:53] * Steki (~steki@bojanka.net) Quit (Quit: Ja odoh a vi sta 'ocete...)
[2:13] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[2:13] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[2:19] * glowell (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[3:12] * maxiz (~pfliu@202.108.130.138) has joined #ceph
[3:37] * mib_mw0bvk (c9024c3f@ircip3.mibbit.com) has joined #ceph
[3:40] <mib_mw0bvk> hi
[3:42] * mib_mw0bvk (c9024c3f@ircip3.mibbit.com) Quit ()
[3:48] * plut0 (~cory@pool-96-236-43-69.albyny.fios.verizon.net) has left #ceph
[4:00] * deepsa (~deepsa@122.172.32.42) has joined #ceph
[4:17] * sjustlaptop (~sam@68-119-138-53.dhcp.ahvl.nc.charter.com) has joined #ceph
[4:43] * glowell (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) has joined #ceph
[4:45] * rweeks (~rweeks@c-24-4-66-108.hsd1.ca.comcast.net) has joined #ceph
[4:47] * maxiz (~pfliu@202.108.130.138) Quit (Remote host closed the connection)
[4:48] * winston-d (~zhiteng@pgdmzpr01-ext.png.intel.com) has joined #ceph
[4:56] * yoshi (~yoshi@p11251-ipngn4301marunouchi.tokyo.ocn.ne.jp) Quit (Remote host closed the connection)
[5:06] * sjustlaptop (~sam@68-119-138-53.dhcp.ahvl.nc.charter.com) Quit (Ping timeout: 480 seconds)
[5:10] * yoshi (~yoshi@p11251-ipngn4301marunouchi.tokyo.ocn.ne.jp) has joined #ceph
[5:17] * maxiz (~pfliu@202.108.130.138) has joined #ceph
[5:27] * Ryan_Lane (~Adium@c-67-160-217-184.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[6:00] * Cube (~Cube@cpe-76-95-223-199.socal.res.rr.com) has joined #ceph
[6:10] * xiaoxi (~xiaoxiche@jfdmzpr02-ext.jf.intel.com) has joined #ceph
[6:20] * glowell (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[6:31] * yoshi (~yoshi@p11251-ipngn4301marunouchi.tokyo.ocn.ne.jp) Quit (Remote host closed the connection)
[6:44] * sjustlaptop (~sam@68-119-138-53.dhcp.ahvl.nc.charter.com) has joined #ceph
[7:02] * yoshi (~yoshi@EM117-55-68-133.emobile.ad.jp) has joined #ceph
[7:18] * lx0 (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[7:20] * lx0 (~aoliva@lxo.user.oftc.net) has joined #ceph
[7:29] * sjustlaptop (~sam@68-119-138-53.dhcp.ahvl.nc.charter.com) Quit (Ping timeout: 480 seconds)
[7:51] * tnt (~tnt@162.63-67-87.adsl-dyn.isp.belgacom.be) has joined #ceph
[7:54] * rweeks (~rweeks@c-24-4-66-108.hsd1.ca.comcast.net) Quit (Quit: ["Textual IRC Client: www.textualapp.com"])
[8:19] * madkiss (~madkiss@chello062178057005.20.11.vie.surfer.at) has joined #ceph
[8:22] * madkiss1 (~madkiss@chello062178057005.20.11.vie.surfer.at) has joined #ceph
[8:27] * madkiss (~madkiss@chello062178057005.20.11.vie.surfer.at) Quit (Ping timeout: 480 seconds)
[8:46] * yoshi (~yoshi@EM117-55-68-133.emobile.ad.jp) Quit (Remote host closed the connection)
[8:54] <xiaoxi> Excuseme, I just wonder how much BW can you achieve for sequential write(i.e 4M request size)?
[8:55] <xiaoxi> I have setuped a 8 node ceph cluser, each node with 3 7200PRM sata +1 SSD for journal.
[8:56] <xiaoxi> But only ~500MB/s BW in 4M sequential write is measured.
[9:06] * loicd (~loic@2a01:e35:2eba:db10:c9da:62f:1fa4:a1bd) has joined #ceph
[9:22] * yoshi (~yoshi@p11251-ipngn4301marunouchi.tokyo.ocn.ne.jp) has joined #ceph
[9:28] * gucki (~smuxi@80-218-125-247.dclient.hispeed.ch) has joined #ceph
[9:28] <gucki> good morning :)
[9:30] <madkiss1> mornin
[9:30] <madkiss1> I've just packaged python-pushy for Debian and Ubuntu.
[9:31] * madkiss1 is now known as madkiss
[9:32] * tnt (~tnt@162.63-67-87.adsl-dyn.isp.belgacom.be) Quit (Read error: Operation timed out)
[9:34] * ScOut3R_ (~ScOut3R@catv-80-98-44-93.catv.broadband.hu) has joined #ceph
[9:35] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has joined #ceph
[9:36] * BManojlovic (~steki@91.195.39.5) has joined #ceph
[9:42] * fc (~fc@home.ploup.net) has joined #ceph
[9:42] * Leseb (~Leseb@193.172.124.196) has joined #ceph
[9:46] <darkfader> xiaoxi: 500MB/s would match your SSD's speed quite well, right?
[9:48] * deepsa (~deepsa@122.172.32.42) Quit (Ping timeout: 480 seconds)
[9:49] * verwilst (~verwilst@d5152FEFB.static.telenet.be) has joined #ceph
[9:49] * deepsa (~deepsa@115.184.72.171) has joined #ceph
[9:51] * tnt (~tnt@212-166-48-236.win.be) has joined #ceph
[9:52] * winston-d (~zhiteng@pgdmzpr01-ext.png.intel.com) Quit (Quit: Leaving)
[9:56] * maxiz (~pfliu@202.108.130.138) Quit (Quit: Ex-Chat)
[10:05] * yanzheng (~zhyan@jfdmzpr03-ext.jf.intel.com) Quit (Remote host closed the connection)
[10:08] * xiaoxi (~xiaoxiche@jfdmzpr02-ext.jf.intel.com) Quit (Ping timeout: 480 seconds)
[10:17] * gregorg (~Greg@78.155.152.6) has joined #ceph
[10:25] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) has joined #ceph
[10:33] * deepsa_ (~deepsa@122.172.173.29) has joined #ceph
[10:35] * BManojlovic (~steki@91.195.39.5) Quit (Quit: Ja odoh a vi sta 'ocete...)
[10:38] * deepsa (~deepsa@115.184.72.171) Quit (Ping timeout: 480 seconds)
[10:38] * deepsa_ is now known as deepsa
[10:39] * BManojlovic (~steki@91.195.39.5) has joined #ceph
[10:44] * maxiz (~pfliu@222.128.136.70) has joined #ceph
[10:49] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) Quit (Quit: tryggvil)
[10:50] * deepsa (~deepsa@122.172.173.29) Quit (Ping timeout: 480 seconds)
[10:51] * deepsa (~deepsa@122.172.208.47) has joined #ceph
[10:52] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) has joined #ceph
[10:54] * iH (~iH@82.166.185.149) has joined #ceph
[11:08] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) Quit (Quit: Leaving.)
[11:10] * match (~mrichar1@pcw3047.see.ed.ac.uk) has joined #ceph
[11:23] * deepsa_ (~deepsa@122.167.174.53) has joined #ceph
[11:23] * deepsa (~deepsa@122.172.208.47) Quit (Ping timeout: 480 seconds)
[11:23] * deepsa_ is now known as deepsa
[11:30] * lx0 is now known as lxo
[11:32] <lxo> I'm getting frequent CDir.cc:1672 assertion failures with want == committed_version with 0.54
[11:32] <jluis> lxo, is that on mds?
[11:32] <lxo> if I relax the assert, the mds log grows unbounded as the stuff in there never gets collected
[11:33] <lxo> yeah
[11:33] <lxo> does this happen to be a known problem? with a known fix?
[11:33] <jluis> looks like some more to keep slang entertained ;)
[11:33] <jluis> not that I know of
[11:33] <jluis> but slang is indeed the right person to ask
[11:34] <jluis> he should be around in 5 hours or so
[11:35] <jluis> lxo, you could also file a bug on the tracker or check with the mailing list
[11:36] <jluis> apparently it's thanksgiving week, and I'm not really sure who's going to be around for that matter, so the list might be the best bet
[11:36] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has joined #ceph
[11:36] * loicd (~loic@2a01:e35:2eba:db10:c9da:62f:1fa4:a1bd) Quit (Quit: Leaving.)
[11:38] <lxo> jluis, I'll dig into it for a bit before resorting to the list
[11:48] * xiaoxi (~xiaoxiche@134.134.139.76) has joined #ceph
[11:55] * loicd (~loic@90.84.144.75) has joined #ceph
[11:57] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has left #ceph
[12:01] * loicd (~loic@90.84.144.75) Quit (Quit: Leaving.)
[12:09] * MikeMcClurg (~mike@cpc10-cmbg15-2-0-cust205.5-4.cable.virginmedia.com) Quit (Quit: Leaving.)
[12:15] * deepsa_ (~deepsa@122.167.175.88) has joined #ceph
[12:16] * deepsa (~deepsa@122.167.174.53) Quit (Ping timeout: 480 seconds)
[12:16] * deepsa_ is now known as deepsa
[12:18] <jtang> so it's possible to use the rados objects and the watch notifications as an alternative to 0mq or resque
[12:21] * sileht (~sileht@sileht.net) Quit (Quit: WeeChat 0.3.8)
[12:24] * sileht (~sileht@sileht.net) has joined #ceph
[12:33] * nhorman (~nhorman@hmsreliant.think-freely.org) has joined #ceph
[12:33] <nhorman> dentist
[12:34] <liiwi> hmm, reminds me that I'll need to make appointment for annual check
[12:36] * yoshi (~yoshi@p11251-ipngn4301marunouchi.tokyo.ocn.ne.jp) Quit (Remote host closed the connection)
[12:37] * calebamiles (~caleb@c-24-128-194-192.hsd1.vt.comcast.net) Quit (Ping timeout: 480 seconds)
[12:41] * iH (~iH@82.166.185.149) Quit ()
[12:49] * MikeMcClurg (~mike@firewall.ctxuk.citrix.com) has joined #ceph
[13:05] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) Quit (Quit: tryggvil)
[13:05] * Cube (~Cube@cpe-76-95-223-199.socal.res.rr.com) Quit (Read error: Operation timed out)
[13:06] * yoshi (~yoshi@p11251-ipngn4301marunouchi.tokyo.ocn.ne.jp) has joined #ceph
[13:09] * ctrl (~Nrg3tik@78.25.73.250) has joined #ceph
[13:14] * mtk (~mtk@ool-44c35983.dyn.optonline.net) has joined #ceph
[13:18] * yoshi (~yoshi@p11251-ipngn4301marunouchi.tokyo.ocn.ne.jp) Quit (Ping timeout: 480 seconds)
[13:28] * mtk (~mtk@ool-44c35983.dyn.optonline.net) Quit (Quit: Leaving)
[13:29] * mtk (~mtk@ool-44c35983.dyn.optonline.net) has joined #ceph
[13:31] * loicd (~loic@magenta.dachary.org) has joined #ceph
[13:36] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[13:37] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[13:43] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) has joined #ceph
[13:45] * tryggvil_ (~tryggvil@rtr1.tolvusky.sip.is) has joined #ceph
[13:49] * calebamiles (~caleb@65-183-137-95-dhcp.burlingtontelecom.net) has joined #ceph
[13:51] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[13:51] * loicd (~loic@magenta.dachary.org) has joined #ceph
[13:51] * itamar_ (~itamar@82.166.185.149) has joined #ceph
[13:51] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) Quit (Ping timeout: 480 seconds)
[13:51] * tryggvil_ is now known as tryggvil
[14:00] * calebamiles (~caleb@65-183-137-95-dhcp.burlingtontelecom.net) Quit (Quit: Leaving.)
[14:07] * maxiz_ (~pfliu@221.216.134.163) has joined #ceph
[14:14] * maxiz (~pfliu@222.128.136.70) Quit (Ping timeout: 480 seconds)
[14:37] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has joined #ceph
[14:40] * elder (~elder@c-71-195-31-37.hsd1.mn.comcast.net) Quit (Ping timeout: 480 seconds)
[14:50] * timmclaughlin (~timmclaug@69.170.148.179) has joined #ceph
[14:57] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[14:58] * loicd (~loic@magenta.dachary.org) has joined #ceph
[14:59] * elder (~elder@c-71-195-31-37.hsd1.mn.comcast.net) has joined #ceph
[14:59] * ChanServ sets mode +o elder
[15:00] * elder (~elder@c-71-195-31-37.hsd1.mn.comcast.net) Quit (Remote host closed the connection)
[15:00] * elder (~elder@c-71-195-31-37.hsd1.mn.comcast.net) has joined #ceph
[15:00] * ChanServ sets mode +o elder
[15:12] * MikeMcClurg (~mike@firewall.ctxuk.citrix.com) Quit (Read error: Connection reset by peer)
[15:12] * MikeMcClurg (~mike@firewall.ctxuk.citrix.com) has joined #ceph
[15:14] * ScOut3R_ (~ScOut3R@catv-80-98-44-93.catv.broadband.hu) Quit (Remote host closed the connection)
[15:15] * deepsa (~deepsa@122.167.175.88) Quit (Read error: Operation timed out)
[15:40] * aliguori (~anthony@cpe-70-123-145-75.austin.res.rr.com) has joined #ceph
[15:44] * deepsa (~deepsa@122.172.159.35) has joined #ceph
[16:03] <gucki> are there any known memory leaks when using latest stable argonaut with kvm? i really think there must be one, because some kvm guests limited to for example 1024 mb ram are consuming around 2000 mb ram (res) after a few days...and these are those which have heavy disk activity.. :(
[16:04] <gucki> probably memory leaks in librados, librdb or the qemu rbd driver? any idea how i could debug this best?
[16:05] * PerlStalker (~PerlStalk@72.166.192.70) has joined #ceph
[16:07] <Meyer__> gucki: Are those guests running with cache=none ?
[16:07] <gucki> Meyer__: no, with writeback 32mb. so i assume a little overhead (50 mb max for libraries, cache etc.), but not 1000 mb...
[16:09] <Meyer__> gucki: Oh you can limit the amount of writeback cache, I was not aware of that
[16:48] * xiaoxi (~xiaoxiche@134.134.139.76) Quit (Remote host closed the connection)
[16:49] * xiaoxi (~xiaoxiche@134.134.139.76) has joined #ceph
[16:56] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) Quit (Quit: tryggvil)
[16:57] * dilemma (~dilemma@2607:fad0:32:a02:1e6f:65ff:feac:7f2a) has joined #ceph
[17:00] * match (~mrichar1@pcw3047.see.ed.ac.uk) Quit (Quit: Leaving.)
[17:01] * BManojlovic (~steki@91.195.39.5) Quit (Quit: Ja odoh a vi sta 'ocete...)
[17:04] * guigouz1 (~guigouz@201.83.213.121) has joined #ceph
[17:05] <slang> sage: can we merge wip-3431?
[17:07] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) has joined #ceph
[17:07] * verwilst (~verwilst@d5152FEFB.static.telenet.be) Quit (Quit: Ex-Chat)
[17:14] <jluis> slang, lxo was hitting an assertion on CDir.cc earlier this morning; have you ever seen it?
[17:17] <gucki> Meyer__: yes, it can be done on the command line to kvm and in the ceph config :)
[17:17] <gucki> Meyer__: at least i hope the switch works :-)
[17:17] <gucki> Meyer__: can i somehow check how much cache it is using?
[17:18] * calebamiles (~caleb@65-183-137-95-dhcp.burlingtontelecom.net) has joined #ceph
[17:18] <slang> jluis: what was the assert?
[17:18] <jluis> sorry, didn't cross my mind to paste it :p
[17:18] <jluis> <lxo> I'm getting frequent CDir.cc:1672 assertion failures with want == committed_version with 0.54
[17:18] <jluis> <jluis> lxo, is that on mds?
[17:18] <jluis> <lxo> if I relax the assert, the mds log grows unbounded as the stuff in there never gets collected
[17:20] <slang> I think that might be fixed in master
[17:20] <jluis> lxo ^^
[17:21] <slang> looking...
[17:25] * xiaoxi (~xiaoxiche@134.134.139.76) Quit (Ping timeout: 480 seconds)
[17:25] <slang> lxo: can you paste/post the full assertion stack trace you get?
[17:25] <dilemma> getting an interesting problem with attaching an RBD volume in libvirt 1.0.0: failed to open file 'pool_name/volume_name': No such file or directory
[17:26] <slang> lxo: on 0.54, there's no assertion at CDir.cc:1672, so I want to make sure I'm looking at the right place
[17:26] <dilemma> when the exact same XML snippet worked with an older version of libvirtd
[17:26] <dilemma> 0.9.13
[17:31] * met (~met@bl18-229-91.dsl.telepac.pt) has joined #ceph
[17:31] * maxiz_ (~pfliu@221.216.134.163) Quit (Ping timeout: 480 seconds)
[17:32] <met> Hi, I'm trying to setup a ceph cluster but after a service restart cluster is stuck in "boot mode", and I can't do any I/O on any pool. Can anyone here help?
[17:40] * itamar_ (~itamar@82.166.185.149) Quit (Remote host closed the connection)
[17:41] * maxiz_ (~pfliu@111.192.248.200) has joined #ceph
[17:46] * tnt (~tnt@212-166-48-236.win.be) Quit (Read error: Operation timed out)
[17:56] * vata (~vata@208.88.110.46) has joined #ceph
[18:04] * tnt (~tnt@162.63-67-87.adsl-dyn.isp.belgacom.be) has joined #ceph
[18:07] <met> I'm getting this message in odd logs: osd.1 10 still in boot state, dropping message osd_op(client.4800.0:1 [pgls start_epoch 0] 3.0) v4
[18:07] <met> "rados -p tt ls" hangs
[18:08] <met> can anyone help me on this isse?
[18:08] <jluis> are your monitors working?
[18:08] <jluis> what does 'ceph -s' return?
[18:10] * MikeMcClurg (~mike@firewall.ctxuk.citrix.com) Quit (Quit: Leaving.)
[18:10] <met> health HEALTH_WARN 8 pgs stuck inactive; 8 pgs stuck unclean
[18:10] <met> monmap e1: 3 mons at {a=172.20.210.148:6789/0,b=172.20.210.150:6789/0,c=172.20.210.146:6789/0}, election epoch 2, quorum 0,1,2 a,b,c
[18:10] <met> osdmap e10: 2 osds: 2 up, 2 in
[18:10] <met> pgmap v456: 392 pgs: 8 creating, 384 active+clean; 0 bytes data, 16455 MB used, 1709 GB / 1725 GB avail
[18:11] <met> mdsmap e1: 0/0/1 up
[18:12] <met> even with the cluster healthy, the problem persists
[18:14] <met> (precise, kernel 3.2, ceph 0.48.2argonaut
[18:14] <jluis> met, do you have all your pg's active and clean?
[18:14] <jluis> I mean, once it's healthy
[18:15] <met> not now, because I was trying to understand why the rados command was hanging
[18:16] <met> I did several installations and every time a do a service ceph -a restart, the cluster never comes up ok, although ceph -s return HEALTH_OK
[18:18] * jlogan1 (~Thunderbi@2600:c00:3010:1:1ccf:467e:284:aea8) has joined #ceph
[18:18] <jluis> afaik, no data will be written to the cluster unless all your pgs are active+clean, at least in a 2 osd scenario
[18:18] <jluis> may be wrong
[18:18] <jluis> sjust, around?
[18:18] <met> Yes, I understand that
[18:19] <jluis> also, pretty awesome to see a *.dsl.telepac.pt in the channel :p
[18:20] <met> jluis: Why?
[18:20] <jluis> I think it's the first time I see a .pt around
[18:21] <met> :)
[18:21] <jluis> other than me, that is
[18:21] <met> a .novis.pt :)
[18:22] <jluis> indeed
[18:22] <met> Are you Portuguse?
[18:22] <met> sorry, Portuguese?
[18:22] <jluis> I sure am
[18:23] <met> Nice!
[18:23] <met> I'm going to wipe /var/lib/ceph, rerun mkcephfs and reproduce the problem. Can you help me debug it?
[18:24] <jluis> as far as I'm able; but I'm sure other's will give you a hand if it comes to that
[18:24] <jluis> *others
[18:24] <met> ok
[18:28] <met> jluis: btw, are you using ceph in production?
[18:28] * bchrisman (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[18:28] <jluis> not really, just dev
[18:30] <met> humm, what do you think?
[18:31] <jluis> what do I think about what?
[18:31] <jluis> ceph?
[18:31] <met> about ceph, of course :-)
[18:33] <jluis> well, best thing that ever happened since the invention of bacon strips
[18:34] <met> me too, if only i could get it to work reliably...
[18:34] <met> the cluster is now scrubbing away...
[18:35] <jluis> most of the times, it's a matter of minor misconfiguration issues
[18:35] <lxo> slang, sorry, my CDir.cc had an additional 9 lines inserted before that assert
[18:35] <lxo> from a patch by sage to restore snapshots (more) properly
[18:36] <lxo> 1: (CDir::commit(unsigned long, Context*, bool)+0x3cd) [0x64072d]
[18:36] <lxo> 2: (LogSegment::try_to_expire(MDS*, C_GatherBuilder&)+0xd93) [0x4d7a13]
[18:36] <lxo> 3: (MDLog::try_expire(LogSegment*)+0x5c) [0x69ab9c]
[18:36] <lxo> 4: (MDLog::_maybe_expired(LogSegment*)+0x217) [0x69b6b7]
[18:36] <lxo> 5: (Context::complete(int)+0xa) [0x4a2dba]
[18:36] <lxo> 6: (C_Gather::sub_finish(Context*, int)+0x2cf) [0x4a475f]
[18:36] <lxo> 7: (C_Gather::C_GatherSub::finish(int)+0x12) [0x4a4802]
[18:36] <lxo> 8: (MDS::_dispatch(Message*)+0x49b) [0x4c451b]
[18:36] <lxo> 9: (MDS::ms_dispatch(Message*)+0x20b) [0x4c5beb]
[18:36] <lxo> 10: (DispatchQueue::entry()+0x6b1) [0x7d29f1]
[18:36] <lxo> 11: (DispatchQueue::DispatchThread::entry()+0xd) [0x75a42d]
[18:37] <lxo> I looked a bit into it before giving up this time, and reverting to an earlier snapshot like I did last time it happened to me
[18:40] <lxo> skipping the commit if the dir already has its current version committed causes the LogSegment destructor to abort because not all of its lists are empty. running the given ->finish crashes when it tries to take a lock (presumably already taken in that thread). letting it repeat the commit (with various changes to make it actually submit the mutation) appears to work, to some extent, but it still doesn't expire old segments
[18:40] <lxo> that bit got me confused, for I don't see why it wouldn't, unlike simply skipping the commit, which does leave waiters waiting
[18:43] <lxo> I didn't look into why the dir gets a commit for an already-committed version. what I did find out after much digging was that the dir was fetched and a commit was issued in response to another request issued to the mds, before it attempted to expire the old segment that requested a commit of the already-committed dir
[18:43] <lxo> however, even after killing all clients, the problem still occurred, so I suspect it was some self-request, possibly caused by pending operations on hard links or somesuch
[18:45] <lxo> I tried to start the 0.53 mds too, but it failed in just the same way. so it's not something new with the recovery, it seems to be something that the new mds leaves in its journal that stops it from recovering successfully, or perhaps some symptom of earlier corruption in the filesystem (tons of hardlinks and snapshots here and there seem to be a sure way to introduce inconsistency in the filesystem metadata :-( that I happen to be tripping over now for som
[18:45] <lxo> e reason
[18:47] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) Quit (Quit: tryggvil)
[18:49] <lxo> slang, does this look like the scenario you said that might be fixed in master?
[18:49] <slang> lxo: sorry - one sec
[18:51] <met> jluis: I've restart the cluster ceph -s returns:
[18:51] <met> health HEALTH_OK
[18:51] <met> monmap e1: 3 mons at {a=172.20.210.148:6789/0,b=172.20.210.150:6789/0,c=172.20.210.146:6789/0}, election epoch 2, quorum 0,1,2 a,b,c
[18:51] <met> osdmap e4: 2 osds: 2 up, 2 in
[18:51] <met> pgmap v219: 384 pgs: 384 active+clean; 0 bytes data, 16455 MB used, 1709 GB / 1725 GB avail
[18:51] <met> mdsmap e1: 0/0/1 up
[18:51] <met> executing "rados -p data ls " hangs...
[18:52] <jluis> met, 'rados --debug-monc 20 --debug-auth 20 -p data ls'
[18:53] <dilemma> I'm not getting much in the #virt channel, so I wonder if someone here is familiar with: I'm upgrading from libvirt v0.9.13 (with rbd support, but without rbd pool support) to v1.0.0 (with support for rbd volumes and pools) and now, due to pool support, it's requiring librbd. I don't need pool support, and don't want librbd, but I do need rbd volume support
[18:53] <met> jluis - more than 50 lines, should i post it here or somewhere else?
[18:54] <jluis> somewhere else
[18:54] <jluis> pastebin, for instance
[18:54] * jluis is now known as joao
[18:55] <met> jluis - http://pastebin.com/ty6W7gS7
[18:58] <met> jluis . it seems an auth problem, right?
[18:58] <joao> not really
[19:00] <joao> it's not an auth problem, and rados is able to contact the monitors
[19:00] <lxo> slang, I'm crashing in bed now; let me know and I'll give it a try when I get back up. thanks
[19:00] <joao> met, this might something dead obvious, but I'm not finding anything wrong
[19:01] <joao> well, not on the monitor front at least
[19:02] <joao> brb; snack before standup
[19:02] <met> the logs on od-0 state:
[19:03] <met> 2012-11-19 18:02:42.611193 7faa6e9ed700 20 osd.0 4 update_osd_stat osd_stat(8227 MB used, 591 GB avail, 599 GB total, peers []/[])
[19:03] <met> 2012-11-19 18:02:42.611203 7faa6e9ed700 5 osd.0 4 heartbeat: osd_stat(8227 MB used, 591 GB avail, 599 GB total, peers []/[])
[19:03] <met> the number of peers is wrong, is it not?
[19:03] <met> also on osd-1 I have:
[19:03] <met> 2012-11-19 18:01:57.528836 7f88cac8f700 7 osd.1 4 still in boot state, dropping message osd_op(client.4211.0:1 [pgls start_epoch 0] 0.0) v4
[19:04] <met> wverytime i do rados ls on a pool
[19:04] <met> wverytime i do rados ls on a pool
[19:04] <met> everytime i do rados ls on a pool
[19:05] * jbd_ (~jbd_@34322hpv162162.ikoula.com) has left #ceph
[19:05] * The_Bishop (~bishop@2001:470:50b6:0:f1c2:7943:e8e2:1587) Quit (Quit: Wer zum Teufel ist dieser Peer? Wenn ich den erwische dann werde ich ihm mal die Verbindung resetten!)
[19:06] <met> what worries me is the: "still in boot state".
[19:06] <met> I think this is why the client hangs.
[19:07] <met> I can paste bin the config
[19:08] * jjgalvez (~jjgalvez@cpe-76-175-17-226.socal.res.rr.com) has joined #ceph
[19:09] * Leseb (~Leseb@193.172.124.196) Quit (Quit: Leseb)
[19:12] * bchrisman (~Adium@108.60.121.114) has joined #ceph
[19:13] <joao> met, I can't say I have seen that happen before
[19:13] <joao> maybe someone else might?
[19:13] * chutzpah (~chutz@199.21.234.7) has joined #ceph
[19:13] * Cube (~Cube@cpe-76-95-223-199.socal.res.rr.com) has joined #ceph
[19:13] <joao> given that ceph -s shows both osds up and in the cluster, I know for sure that both osds have informed the monitors that they have booted
[19:14] <joao> but I may be missing something
[19:14] <met> yes, but they're still in boot mode, and thus refusing connections
[19:15] <met> osd 0 - heartbeat: osd_stat(8227 MB used, 1117 GB avail, 1125 GB total, peers []/[])
[19:16] <met> odd 1 - heartbeat: osd_stat(8227 MB used, 591 GB avail, 599 GB total, peers []/[])
[19:16] <met> they're not in sync!
[19:16] <met> they're only "seeing" the disks they own...
[19:17] <met> but no errors...
[19:17] <slang> sage: I think I spoke too soon on wip-3431
[19:17] <met> ceph -s still returns: health HEALTH_OK
[19:18] <joao> met, hang in in there; we're having the standup and I've already asked sjust to give you a hand as soon as he can
[19:18] <met> ok, thanks
[19:22] <sjust> met: can you give me the output of ceph osd tree, ceph pg dump, ceph osd dump?
[19:22] <met> sjust: sure, just a moment
[19:24] * rweeks (~rweeks@c-98-234-186-68.hsd1.ca.comcast.net) has joined #ceph
[19:25] <met> sjust: http://pastebin.com/8qZV0xau
[19:26] <sjust> according to this, both osds are up/in, all pgs are healthy
[19:27] <sjust> the hb in/hb out fields are a bit of an anomaly
[19:29] <met> what's the meaning of hb?
[19:29] <joao> heartbeat
[19:30] <joao> between the osd and the monitor, and iirc between osds
[19:30] <sjust> can you mark osd 0 out and then back in?
[19:30] <sjust> ceph osd out 0; ceph osd in 0
[19:30] <joao> in that case, I think it's just the heartbeat between the monitors and the osds
[19:30] <sjust> you should see the pgs go from active clean to peering back to active-clean at some point
[19:31] <sjust> trying to verify that the monitors are working
[19:31] <slang> lxo: I don't think its fixed in master
[19:32] <slang> lxo: sorry - I was looking at your assertion line: want == committed_version
[19:32] <slang> lxo: which is different from master, but must be part of the patch that sage sent you
[19:32] <slang> lxo: was that patch posted to the mailing list? (I'm having trouble finding it)
[19:32] <met> sjust: done it, but pgs never changed state
[19:33] <sjust> ok
[19:33] <met> according to the logs, all monitors are ok
[19:33] <sjust> can you post ceph pg dump again?
[19:34] <met> sjust: http://pastebin.com/v2sJw4h8
[19:35] * gregaf (~Adium@2607:f298:a:607:b01b:ef4c:9fb6:39c9) Quit (Quit: Leaving.)
[19:37] * gregaf (~Adium@2607:f298:a:607:b4be:20f0:4787:7dfc) has joined #ceph
[19:38] <sjust> met: can you post the mon.a log?
[19:40] <met> http://pastebin.com/15WkWMuh
[19:41] <met> shall i post the config file?
[19:41] <sjust> ok, can you reboot mon.a with debug ms = 1 and debug mon = 20 and osd0 with debug ms = 1 and debug osd = 20
[19:43] <met> done
[19:43] <sjust> after confirming that it's still misbehaving, post the osd0 log and the mon.a log as well as the ceph.conf from each node
[19:44] <sjust> (that is, the ceph.conf from the mon.a node and the osd.0 node0
[19:44] <sjust> I suspect that the osd is unable to connect properly to the mon
[19:44] <sjust> the osd will sit in boot state until the monitor cluster tells it to continue
[19:45] <joao> the weird thing is that the monitor is stating both osds as up and running
[19:45] <sjust> yeah, I'm not completely clear on how that can happen, but it seems to be the case that neither osd is talking to the mons
[19:46] <gregaf> this is after rebooting, right?
[19:46] <met> ceph.conf: http://pastebin.com/FL8CLyX0
[19:46] <gregaf> so the OSDs connected the first time but can't connect on startup, and the monitor hasn't detected them as down yet due to the long timeout?
[19:46] <sjust> I mean, the mons aren't getting failure messages, but they are supposed to require stat updates from time to time, right?
[19:46] <met> gregaf: rebooting or just restarting ceph
[19:46] <gregaf> sjust: that's like 15 minutes thoug
[19:46] <gregaf> *though
[19:46] <sjust> gregaf: ah
[19:48] * davidz (~Adium@ip68-96-75-123.oc.oc.cox.net) has joined #ceph
[19:49] <joao> well, the osds must be delivering at least the MOSDBoot message, as well as MPGStats
[19:49] <joao> or must have at least once
[19:49] <met> mon-a log, http://pastebin.com/iTPscsKb
[19:50] <met> osd-0 log, http://pastebin.com/DekHwDf8
[19:51] <joao> met, we need the leader's log; that would be mon.c
[19:52] * The_Bishop (~bishop@p4FCDF0D3.dip.t-dialin.net) has joined #ceph
[19:54] <met> I've injects the debug flags into the cluster
[19:54] <met> * injected
[19:55] <met> mon-c lot, http://pastebin.com/SwQxu7RV
[19:56] * fc (~fc@home.ploup.net) Quit (Quit: leaving)
[19:59] <sjust> I need to see the osd0 start up, can you add debug osd = 20 and debug ms = 1 to osd0's config and restart it?
[20:00] <met> ok
[20:03] <joao> fwiw, the log shows osd.0 subscription and pg stats messages
[20:03] <sjust> yeah, I know
[20:03] * Ryan_Lane (~Adium@216.38.130.167) has joined #ceph
[20:03] <joao> didn't find any mentions from osd.1 though
[20:03] <sjust> very odd
[20:04] <joao> might it be possible to be some firewall in place?
[20:04] <joao> allowing outbound connections from the osds but no inbound connections?
[20:04] <sjust> maybe
[20:04] <joao> met, ^^
[20:05] <met> osd-0 log, http://pastebin.com/vVZgFnFW
[20:05] <met> sorry for the delay
[20:05] <met> no firewall active
[20:06] <sjust> let it run for a few minutes and then repost the log, it's still reading its ondisk state
[20:06] <met> ok
[20:07] <met> odd-0 is now in "tick mode"
[20:07] <met> what are you interested in?
[20:07] <sjust> ok, repost?
[20:07] <sjust> I'm not sure yet, but I want to see it receive an osd map
[20:09] <met> http://pastebin.com/s9QkMk1E
[20:10] * tryggvil (~tryggvil@17-80-126-149.ftth.simafelagid.is) has joined #ceph
[20:13] <met> may be this is what you're looking for? http://pastebin.com/QbkTXg6H
[20:13] <sjust> can you gzip the entire log and put it on cephdrop@ceph.com?
[20:13] <met> ok
[20:13] * Ryan_Lane (~Adium@216.38.130.167) Quit (Quit: Leaving.)
[20:14] * Ryan_Lane (~Adium@216.38.130.167) has joined #ceph
[20:16] <sjust> met: check that, the bug is on the monitor side
[20:16] <sjust> can you include the gzipped logs from the mons?
[20:17] <met> all the mons or just the leader?
[20:17] <sjust> all
[20:17] <elder> joshd, standup
[20:18] * dilemma (~dilemma@2607:fad0:32:a02:1e6f:65ff:feac:7f2a) Quit (Quit: Leaving)
[20:19] * tryggvil (~tryggvil@17-80-126-149.ftth.simafelagid.is) Quit (Quit: tryggvil)
[20:21] <met> got all the logs, upload via ftp?
[20:21] <sjust> sftp, yeah
[20:24] <met> it's asking for a password?
[20:25] <sjust> met: pm'd you the password
[20:26] <met> pm'd ?
[20:26] <sjust> pass is asdf
[20:27] <met> done, file is min-logs.tgz
[20:27] <sjust> cool
[20:27] <met> mon-logs.tgz
[20:27] <met> (sorry)
[20:28] <sjust> can you check ceph-osd --version and ceph-mon --version?
[20:29] * dmick (~dmick@2607:f298:a:607:8d34:7c9a:3476:1a1e) has joined #ceph
[20:30] <met> ceph version 0.48.2argonaut (commit:3e02b2fad88c2a95d9c0c86878f10d1beb780bfe)
[20:30] <sjust> both?
[20:30] <met> heap, and on all machines
[20:30] <met> (nodes)
[20:31] <sjust> ok, I need the osd.0.log after all
[20:31] <met> sftp?
[20:31] <sjust> yah
[20:31] <sjust> or is it already there?
[20:31] <met> nope
[20:33] <met> ceph-osd.0.log.gz uploaded
[20:37] * tryggvil (~tryggvil@17-80-126-149.ftth.simafelagid.is) has joined #ceph
[20:39] <sjust> met: seems to be mon internal state bug, joao's looking into it
[20:39] <met> ok
[20:41] * noob2 (a5a00214@ircip1.mibbit.com) has joined #ceph
[20:41] <met> Do you have any idea on what's causing it? is it for having onoy 2 osd's?
[20:41] <met> *only
[20:41] <noob2> does anyone know if dream objects backs up their cloud storage onto tape or replicates it off site?
[20:42] * adjohn (~adjohn@69.170.166.146) has joined #ceph
[20:52] * nhm (~nh@184-97-251-146.mpls.qwest.net) has joined #ceph
[20:54] <nhm> good afternoon #ceph
[20:58] <joao> hello nhm
[21:05] <joao> met, do you still have problems with rados?
[21:05] <met> yes, radios -p data ls, just hangs
[21:05] <met> *rados
[21:07] <met> and odd-1 logs: still in boot state, dropping message osd_op(client.4340.0:1 [pgls start_epoch 0] 0.0) v4
[21:07] <joao> but osd0 is no longer in that state
[21:07] <joao> not from the log I just looked into
[21:07] <met> the log comes from osd-1
[21:08] <joao> could you upload that log as well?
[21:10] <met> done: ceph-osd.1.log.gz
[21:10] <joao> ty
[21:10] <met> it's big, very big :-)
[21:11] <joao> 1MB?
[21:11] <joao> maybe I got the wrong one...
[21:12] <joao> doesn't look like it
[21:12] <joao> oh, decompressed that is
[21:12] <joao> okay, not that big either way :)
[21:13] <met> 1.2M compressed, 19M flat
[21:13] <met> I see you're used to big logs files :-)
[21:15] * jjgalvez (~jjgalvez@cpe-76-175-17-226.socal.res.rr.com) Quit (Quit: Leaving.)
[21:16] * Leseb (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) has joined #ceph
[21:17] <dmick> met: Yeah, it's all about Big Data :)
[21:18] <met> :D
[21:18] * The_Bishop (~bishop@p4FCDF0D3.dip.t-dialin.net) Quit (Quit: Wer zum Teufel ist dieser Peer? Wenn ich den erwische dann werde ich ihm mal die Verbindung resetten!)
[21:25] <madkiss> anyone interested in ceph-deploy debs?
[21:25] <madkiss> all along with python-pushy debs?
[21:26] <darkfader> if noone else says yes then i'll do it
[21:27] <madkiss> i'll make them available as soon as the final test is done
[21:27] <madkiss> I did upload python-pushy to Debian NEW already
[21:28] <madkiss> so that's done, I just need to finish on ceph-deploy.
[21:29] <darkfader> it seems that will also bring LIO support to debian
[21:29] <madkiss> hu? who exactly?
[21:29] <darkfader> i'm doing my storage class again in 2 weeks, maybe i should switch to debian instead of fc16
[21:29] <darkfader> madkiss: i just googled and saw there is a lio-utils package now
[21:30] <darkfader> i need an OS that as working ceph and lio-utils which is always an interesting challenge
[21:31] <darkfader> but ceph-deploy packages puts in weight for debian :)
[21:31] * jjgalvez (~jjgalvez@12.248.40.138) has joined #ceph
[21:32] * Anonymous (~Anonymous@adsl-74-248-47-7.pfn.bellsouth.net) has joined #ceph
[21:32] <Anonymous> yo
[21:32] <dmick> oy
[21:34] * Cube (~Cube@cpe-76-95-223-199.socal.res.rr.com) Quit (Quit: Leaving.)
[21:35] <madkiss> darkfader: i expect to have them by tomorrow, maybe even by today
[21:36] <darkfader> i swear i'll open bug reports for anything we find during the class :)
[21:38] <joao> met, still around?
[21:38] <met> yeah
[21:38] <joao> fwiw, this is a long shot and since sjust must be having lunch I haven't run it through him yet
[21:39] <met> shoot anyways
[21:39] <joao> could you try something like 'ceph osd create', and then follow it with 'ceph osd rm osd.2' at your own discretion?
[21:39] <joao> I don't really want you to add another osd, just generate a new osdmap epoch
[21:39] <met> ok
[21:40] <joao> and after that's all done, wait a couple of minutes and upload osd.1's log again
[21:40] * The_Bishop (~bishop@p4FCDF0D3.dip.t-dialin.net) has joined #ceph
[21:41] <joao> and after this I should really go for some dinner
[21:41] <joao> btw, met, where exactly are you from?
[21:41] <met> Porto, actually Maia
[21:41] * Anonymous (~Anonymous@adsl-74-248-47-7.pfn.bellsouth.net) Quit (Quit: Leaving)
[21:41] <joao> cool
[21:42] * madkiss (~madkiss@chello062178057005.20.11.vie.surfer.at) Quit (Read error: Connection reset by peer)
[21:42] <met> I take it you'll only need the just before this commands, right?
[21:42] <joao> it's a bummer though; I was hoping ceph had started rubbing on some Lisbon folks
[21:42] <met> :-)
[21:42] <joao> before and after
[21:43] <joao> but you can upload the whole log and I'll just look for what I need by timestamp
[21:43] <met> ok
[21:43] * noob2 (a5a00214@ircip1.mibbit.com) Quit (Quit: http://www.mibbit.com ajax IRC Client)
[21:44] <met> uploaded: ceph-osd.1.log.gz
[21:45] <joao> thanks
[21:46] <met> you're very welcome!
[21:46] * guigouz1 (~guigouz@201.83.213.121) Quit (Quit: Computer has gone to sleep.)
[21:49] <met> joao: creating more osd's (different machines) will help avoiding this kind of problems?
[21:49] <dmick> sage, sagewk: just hit FAILED assert(p->first == end - p->second.second) again
[21:49] <joao> met, not really
[21:50] <joao> I'm just chasing a theory
[21:50] <dmick> (3428 that is)
[21:51] * madkiss (~madkiss@chello062178057005.20.11.vie.surfer.at) has joined #ceph
[21:51] <madkiss> christ! OS X madness
[21:52] * Cube (~Cube@12.248.40.138) has joined #ceph
[21:53] <met> joao, this is really weird, i've started from scratch half a dozen times. The cluster intializes correctly but never recovers from a simple restart? I mean it's gotta be something with my config, right?
[21:54] <met> although i can't find anything wrong with it...
[21:57] <joao> met, can you please run 'ceph osd getmap -o osd.map' and upload that file?
[22:00] <met> on any node?
[22:01] <met> it's just a single line:
[22:01] <met> got osdmap epoch 8
[22:01] <met> sorry
[22:01] <joao> but it did create a file, right? :)
[22:02] <met> heap, i think i need a coffee :\
[22:03] <met> uploaded: osd.map
[22:08] * adjohn (~adjohn@69.170.166.146) Quit (Quit: adjohn)
[22:09] * MarkN (~nathan@142.208.70.115.static.exetel.com.au) has joined #ceph
[22:10] * MarkN (~nathan@142.208.70.115.static.exetel.com.au) has left #ceph
[22:12] <zynzel> 2logch-t -o -p
[22:12] <zynzel> wrong window :)
[22:13] <madkiss> hello zynzel :)
[22:13] <zynzel> hi!
[22:15] * madkiss (~madkiss@chello062178057005.20.11.vie.surfer.at) Quit (Quit: Leaving.)
[22:16] * madkiss (~madkiss@chello062178057005.20.11.vie.surfer.at) has joined #ceph
[22:21] <met> joao: are you still here?
[22:22] * calebamiles (~caleb@65-183-137-95-dhcp.burlingtontelecom.net) Quit (Remote host closed the connection)
[22:25] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[22:25] * loicd (~loic@magenta.dachary.org) has joined #ceph
[22:31] * nhorman (~nhorman@hmsreliant.think-freely.org) Quit (Quit: Leaving)
[22:34] <lurbs> How does one start to debug what's causing a slow request, specifically one that never seems to complete even after a day or so.
[22:40] <gregaf> lurbs: that's pretty odd; are all your PGs healthy?
[22:42] <lurbs> Reasonably so.
[22:42] <lurbs> http://paste.nothing.net.nz/b5744b
[22:43] <sjust> met: he's grabbing some food
[22:43] <cypher497> what are most people running for linux distributions?
[22:43] <lurbs> Although I'm not sure why it was stuck on active+clean+scrubbing. Didn't come right until I stopped/started osd 4.
[22:43] <gregaf> lurbs: that's 60 seconds old, which is certainly taking longer than it should but is not a day old
[22:43] <gregaf> do you perhaps mean that you've had a slow request warning for a day?
[22:44] <met> sjust: ok, I'm gonna go and eat something to. I'll be back in 30 minutes.
[22:44] <lurbs> That's when it started - more recent is:
[22:44] <lurbs> 2012-11-20 10:02:05.875142 osd.4 192.168.30.10:6812/5546 95662 : [WRN] 1 slow requests, 1 included below; oldest blocked for > 51777.189671 secs
[22:44] <lurbs> 2012-11-20 10:02:05.875146 osd.4 192.168.30.10:6812/5546 95663 : [WRN] slow request 51777.189671 seconds old, received at 2012-11-19 19:39:08.685450: osd_op(client.7567.0:4731 rbd_data.1b1344bae3b8.0000000000000222 [write 196608~3997696] 2.9c669bb8 snapc e=[e]) v4 c
[22:44] <met> sjust: Do you think we can continue to debug this problem?
[22:44] <lurbs> urrently delayed
[22:44] <gregaf> lurbs: ah
[22:44] <lurbs> 2012-11-20 10:02:08.611457 mon.0 192.168.30.10:6789/0 43103 : [INF] pgmap v326734: 3648 pgs: 3644 active+clean, 4 active+clean+scrubbing; 27578 MB data, 57101 MB used, 16683 GB / 16738 GB avail
[22:44] <sjust> yeah, probably
[22:44] <gregaf> what's this cluster used for?
[22:44] <gregaf> and do you have any symptoms of IO not completing?
[22:44] <lurbs> RBD as backing for KVM instances.
[22:45] <gregaf> my inclination here is to go to sjust and ask if this looks like a bug, because I think I remember hearing about one like this :)
[22:45] <lurbs> I wasn't able to get into the VM that was affected until I'd restarted osd 4.
[22:45] <lurbs> Yeah, I see a similar looking sort of thing on the mailing list.
[22:46] * Cube (~Cube@12.248.40.138) Quit (Ping timeout: 480 seconds)
[22:46] <sjust> lurbs: ceph version?
[22:46] * tryggvil (~tryggvil@17-80-126-149.ftth.simafelagid.is) Quit (Quit: tryggvil)
[22:46] <lurbs> 0.54 from debian-testing, on 12.04 LTS.
[22:46] <lurbs> osds are backed by XFS, with journal on SSD.
[22:48] <sjust> ceph pg dump
[22:48] <sjust> ?
[22:49] <lurbs> Is huge. You want the whole thing in a pastebin?
[22:49] <sjust> yes
[22:51] <lurbs> http://paste.nothing.net.nz/0d67e5
[22:51] <lurbs> That's after I restarted osd 4 though, so the hung request is gone.
[22:51] <sjust> ah
[22:52] <lurbs> ...and it truncated it anyway.
[22:52] <sjust> if the hung request was caused by a hung scrub, than it's not surprising that restarting osd.4 made it go away
[22:52] <sjust> unfortunately
[22:52] <sjust> were you seeing hung io?
[22:52] * Cube (~Cube@12.248.40.138) has joined #ceph
[22:52] <lurbs> To a VM, yes. Couldn't get into it to look at exactly what was happening though.
[22:54] <sjust> so the vm was hung?
[22:54] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has left #ceph
[22:54] <lurbs> Yeah. Wouldn't ping, couldn't get a console.
[22:54] <sjust> k
[22:55] <lurbs> Came right, without a reboot, when osd 4 was restarted though.
[22:55] <sjust> right
[22:55] <sjust> I'll need to catch it in the act
[22:56] * nwatkins (~Adium@soenat3.cse.ucsc.edu) has joined #ceph
[22:56] <gregaf> sjust: so this is probably a new scrub bug?
[22:56] <sjust> or something else, no way to know without logs
[22:56] <gucki> are there any known memory leaks when using latest stable argonaut with kvm? i really think there must be one, because some kvm guests limited to for example 1024 mb ram are consuming around 2000 mb ram (res) after a few days...and these are those which have heavy disk activity.. :(
[22:56] <gucki> probably memory leaks in librados, librdb or the qemu rbd driver? any idea how i could debug this best?
[22:57] <joao> back
[22:57] <sjust> gucki: joshd might know
[22:57] <gucki> i'm using qemu-rbd with 32mb writeback cache. so i assume a little overhead (50 mb max for libraries, cache etc.), but not 1000 mb...
[22:57] <lurbs> If it happens again then I should have a look at the running requests via various 'ceph --admin-daemon $socket' commands?
[22:57] <gucki> sjust: is joshd here? :)
[22:58] <sjust> he will be
[22:58] <nwatkins> gregaf: I've got the clock sync bug reproduced now. I'd like to try to verify the problem by forcing the client to refresh its cached inode. Any easy hack to make this happen? It only needs to work for stat()
[22:59] <gregaf> nwatkins: how is refreshing the inode supposed to help?
[22:59] <gregaf> on which node?
[22:59] <nwatkins> gregaf: the idea was it would grab the mtime that the mds sees.
[22:59] <sjust> lurbs: yeah
[23:00] <sjust> it would help immensely if I had osd logs from when the offending request started though
[23:00] <gregaf> so you want the writer to get what the MDS sees before reporting it to the others, right?
[23:00] <sjust> (debug osd = 20, debug filestore = 20, debug ms = 1)
[23:00] <nwatkins> gregaf: yes
[23:00] <lurbs> I'll change the config, and see if I can catch it next time. Thanks!
[23:00] <lurbs> Have to run now, though.
[23:03] <madkiss> does ceph-deploy have any usable version number right now? :D
[23:03] <nwatkins> gregaf: I think this email from Sage a while back is a patch to accomplish this hack: http://marc.info/?l=ceph-devel&m=133178637520337&w=2
[23:03] <gucki> joshd: yt? :)
[23:05] <gregaf> nwatkins: yes, that should do it
[23:05] <gregaf> sorry, trying to reload this into my brain :)
[23:06] <nwatkins> gregaf: :) thanks
[23:06] <gregaf> madkiss: right now ceph-deploy is just part of the main ceph repo since they're evolving in parallel
[23:06] * aliguori (~anthony@cpe-70-123-145-75.austin.res.rr.com) Quit (Remote host closed the connection)
[23:06] <madkiss> i see
[23:06] <gregaf> nwatkins: and was getting really confused since I missed that the skip-the-mds-request logic was so short and stuck at the beginning of the function
[23:07] <madkiss> gregaf: i was thinking about building debian packages for it, and packages need a version number … ;)=
[23:07] <gregaf> madkiss: if you were going for that, then probably matching the Ceph version number it comes from would be a good choice
[23:08] <madkiss> 0.54 at the moment?
[23:08] <gregaf> although I think ceph-deploy is packaged alongside Ceph, isn't it?
[23:08] <madkiss> i haven't found packages of it
[23:08] <Robe> hrm - i'm wondering - what are the biggest installations these days, osdcount-wise?
[23:09] <gregaf> I mean I thought it was in the Ceph packages in recent versions
[23:09] <gregaf> if not, then yes, matching the Ceph version for the repo it comes from would be a good plan
[23:09] <madkiss> gregaf: up to earlier this morning (CET time), python-pushy was not packaged for debian
[23:09] <gregaf> Robe: ~900 daemons in DreamObjects
[23:09] <Robe> ha!
[23:09] <Robe> so I could win the european price
[23:10] <gregaf> possibly? I don't know what the second-largest install count is :)
[23:10] <Robe> looking at 2700 and 5000 disks respectively
[23:10] <madkiss> looking for trouble again, he?
[23:10] <Robe> probably
[23:10] <madkiss> I would have thought that one raid is enough :P
[23:11] <Robe> but first I need to get bcache into something that resembles deterministic behavior
[23:11] <madkiss> anyway
[23:11] <madkiss> tty people later
[23:11] <Robe> bye :)
[23:12] * adjohn (~adjohn@69.170.166.146) has joined #ceph
[23:12] <nhm> Robe: bcache?
[23:12] * nhm perks up
[23:12] <Robe> nhm: yesh.
[23:12] <nhm> Robe: Are you using it with Ceph yet?
[23:12] <Robe> I want to use it as L2ARC for flat files in a local cache fs
[23:12] <Robe> nope
[23:12] * xiaoxi (~xiaoxiche@134.134.139.76) has joined #ceph
[23:13] <nhm> ah, ok
[23:13] <Robe> though I expect that it'll work similarly
[23:13] <nhm> I've been thinking about trying it out for OSDs.
[23:13] <joshd> gucki: I'm not aware of any, but it's possible some were fixed and not backported
[23:13] <Robe> thing is that the documentation is a bit shoddy and the knobs and gauges are a bit hard to interpret
[23:13] <joshd> gucki: the ideal way to debug is using valgrind's massif tool, it gives a trace of exactly what's using memory
[23:13] <gregaf> Robe: that many disks would certainly make us want to be friends :)
[23:14] <joshd> gucki: I'm not sure how well it works against qemu/kvm though
[23:14] <gucki> joshd: ok, but valgrind makes it really slow, right? can i just start kvm using valgrind
[23:14] <nhm> Robe: oh wow, that's a lot of OSDs.
[23:14] <nhm> Robe: Mind if I ask who that would be for?
[23:14] <Robe> gregaf: We'll probably do a small scale testrun and hit you guys up WRT best practices for this scale
[23:15] <Robe> nhm: let's call it "large object high volume CDN"
[23:15] * scalability-junk (~stp@188-193-211-236-dynip.superkabel.de) Quit (Ping timeout: 480 seconds)
[23:15] <rweeks> large media objects? ;)
[23:15] <joshd> gucki: yeah, it'll be a lot slower. try 'valgrind --tool=massif kvm ...'
[23:15] <Robe> it's all opaque to us!
[23:15] <rweeks> ahh
[23:16] <joshd> gucki: it'll give an output file called massif.out.pid
[23:16] <Robe> rweeks: but yeah, gives you the chance to play with infrastructures which are not quite in the trivial space anymore
[23:17] <rweeks> understood
[23:17] <nhm> Robe: I'm not sure how many non-trivial offers I can take. ;)
[23:18] <Robe>
[23:18] <Robe> hehe
[23:18] * nhm looks at his todo list and cries
[23:19] <Robe> todo bankruptcy?
[23:19] <rweeks> todo zero
[23:20] <nhm> That's what happens after I go insane!
[23:20] <gucki> joshd: ok thanks, i'll try the next few days and create a bug report if i find anything reproduceable :)
[23:20] <nhm> granted, it's questionable if any of us are particularly sane anyway, so I guess the point is moot.
[23:21] <Robe> nhm: distributed systems combined with storage combined with database semantics
[23:21] <joshd> gucki: thanks. you might try the 'next' branch too, as it will become bobtail in a couple weeks
[23:21] <Robe> I can't tell what's real anymore these days
[23:22] <gucki> joshd: hopefully i find some leaks (otherwise i've no clue why some guests take that much memory) and it'll be (or already is) fixed in bobtail :-)
[23:23] <met> joao: didi you manage to discover any problem with osd-1 ?
[23:24] * xiaoxi (~xiaoxiche@134.134.139.76) Quit (Remote host closed the connection)
[23:25] <joao> I did manage to find something interesting, but not sure yet
[23:25] <met> like what?
[23:25] <sjust> met,joao: the problem seems to be that osd.0 is starting with a nonce of 0
[23:26] <sjust> met: ubuntu?
[23:26] <met> yeah, 12.04 LTS
[23:26] <met> latest patches applied
[23:27] * slang (~slang@ace.ops.newdream.net) Quit (Ping timeout: 480 seconds)
[23:28] <gregaf> met: do you have some security modules running or anything?
[23:29] <met> i don't think so, but i'll check
[23:29] <met> like apparmor?
[23:29] <sjust> he's asking because gitpid() appears to be returning 0
[23:29] <dmick> 0 is an odd answer
[23:30] <sjust> even, actually
[23:30] <elder> Is that "getpid()" written with a New Orleans accent?
[23:30] <dmick> :-P
[23:30] <sjust> it is
[23:30] <dmick> 1) I can't imagine the security model that hides your own pid
[23:30] <elder> I could hear the twang from here.
[23:30] <sjust> heh
[23:30] <dmick> 2) if it did, surely it would be an error
[23:30] <sjust> yeah, I'm with you
[23:34] * BManojlovic (~steki@212.69.20.139) has joined #ceph
[23:35] <gregaf> met: can you pastebin your ceph.conf for the OSDs?
[23:36] <met> sjust: I've disabled apparmor and rebooted just in case
[23:38] <gregaf> I actually see a few other ways this could happen
[23:38] <met> gregaf: http://pastebin.com/UD8zCfHJ
[23:39] <gregaf> met: is it important that the OSDs bind to those specific ports?
[23:39] <met> not really
[23:40] <gregaf> if not, you can remove the ports from those IPs and that will fix your problem
[23:40] <gregaf> there's a bug which I'm having difficulty tracking the origin of, so I don't want to spin up an insta-fix
[23:40] <gregaf> but I see what's happening here
[23:40] <met> ok. I'll try
[23:43] <met> still not working
[23:43] <met> message:
[23:43] <met> Starting Ceph mon.a on c2virt08...
[23:43] <met> WARNING: 'mon addr' config option 172.20.210.148:0/0 does not match monmap file
[23:43] <met> continuing with monmap configuration
[23:43] <met> starting mon.a rank 1 at 172.20.210.148:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid 6630af05-5f47-437f-a0fe-e42772521483
[23:44] <gregaf> met: oh, sorry, the monitors need them specified
[23:44] <gregaf> I meant just the OSDs :)
[23:44] <met> ok
[23:47] <met> It's working !
[23:47] <met> :D
[23:48] <met> gregaf: what's wrong with specifying the ports for osds ?
[23:48] <gregaf> a bug in our code
[23:48] <gregaf> :)
[23:48] <gregaf> I think it's a trivial fix but I don't have time to check it in detail right now and we're going to want to do so
[23:48] <gregaf> see http://tracker.newdream.net/issues/3510
[23:49] <met> i see
[23:50] <met> Thank you all for solving this issue. I've been trying to find it for over 2 days… :-(
[23:51] <met> btw, what do you recommend for production, 0.48 or 0.54?
[23:57] * jlogan1 (~Thunderbi@2600:c00:3010:1:1ccf:467e:284:aea8) Quit (Ping timeout: 480 seconds)
[23:59] * paravoid (~paravoid@scrooge.tty.gr) Quit (Read error: Connection reset by peer)

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.