#ceph IRC Log

Index

IRC Log for 2012-06-26

Timestamps are in GMT/BST.

[0:03] * plastics- (~plastics@c-69-138-42-222.hsd1.tn.comcast.net) has joined #ceph
[0:03] * plastics (~plastics@c-69-138-42-222.hsd1.tn.comcast.net) Quit (Read error: Connection reset by peer)
[0:10] * widodh_ (~widodh@minotaur.apache.org) has joined #ceph
[0:10] * MapspaM (~clint@xencbyrum2.srihosting.com) has joined #ceph
[0:10] * Azrael_ (~azrael@terra.negativeblue.com) has joined #ceph
[0:10] * Solver_ (~robert@atlas.opentrend.net) has joined #ceph
[0:10] * rosco_ (~r.nap@188.205.52.204) has joined #ceph
[0:11] * acaos_ (~zac@209-99-103-42.fwd.datafoundry.com) has joined #ceph
[0:11] * s[X] (~sX]@eth589.qld.adsl.internode.on.net) Quit (resistance.oftc.net oxygen.oftc.net)
[0:11] * sdouglas (~sdouglas@c-24-6-44-231.hsd1.ca.comcast.net) Quit (resistance.oftc.net oxygen.oftc.net)
[0:11] * mkampe (~markk@aon.hq.newdream.net) Quit (resistance.oftc.net oxygen.oftc.net)
[0:11] * widodh (~widodh@minotaur.apache.org) Quit (resistance.oftc.net oxygen.oftc.net)
[0:11] * SpamapS (~clint@xencbyrum2.srihosting.com) Quit (resistance.oftc.net oxygen.oftc.net)
[0:11] * acaos (~zac@209-99-103-42.fwd.datafoundry.com) Quit (resistance.oftc.net oxygen.oftc.net)
[0:11] * edwardw`away (~edward@ec2-50-19-100-56.compute-1.amazonaws.com) Quit (resistance.oftc.net oxygen.oftc.net)
[0:11] * Azrael (~azrael@terra.negativeblue.com) Quit (resistance.oftc.net oxygen.oftc.net)
[0:11] * rosco (~r.nap@188.205.52.204) Quit (resistance.oftc.net oxygen.oftc.net)
[0:11] * Solver (~robert@atlas.opentrend.net) Quit (resistance.oftc.net oxygen.oftc.net)
[0:12] * s[X]_ (~sX]@eth589.qld.adsl.internode.on.net) has joined #ceph
[0:12] * edwardw`away (~edward@ec2-50-19-100-56.compute-1.amazonaws.com) has joined #ceph
[0:12] * sdouglas (~sdouglas@c-24-6-44-231.hsd1.ca.comcast.net) has joined #ceph
[0:14] * mkampe (~markk@aon.hq.newdream.net) has joined #ceph
[0:17] * sjustlaptop1 (~sam@aon.hq.newdream.net) Quit (Quit: Leaving.)
[0:18] * sjustlaptop (~sam@aon.hq.newdream.net) has joined #ceph
[0:20] * Azrael_ is now known as Azrael
[0:23] * lotreck (~Adium@c-75-75-33-53.hsd1.va.comcast.net) has joined #ceph
[0:29] <MarkS> I did see ceph is not yet ready for production, is there an idea when it is production ready? (I'm thinking about ceph for a backup platform)
[0:29] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[0:30] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) Quit (Quit: Leaving.)
[0:30] * sjustlaptop (~sam@aon.hq.newdream.net) Quit (Read error: Operation timed out)
[0:33] * sagelap (~sage@ace.ops.newdream.net) has joined #ceph
[0:35] <Tv_> MarkS: RADOS, RBD and radosgw are ready (depending on your definition of "production"; use individual consideration etc, Inktank will happily help in evaluating properly)
[0:40] * plastics- (~plastics@c-69-138-42-222.hsd1.tn.comcast.net) Quit (Ping timeout: 480 seconds)
[0:44] * cattelan (~cattelan@2001:4978:267:0:21c:c0ff:febf:814b) has joined #ceph
[0:48] * sjustlaptop (~sam@aon.hq.newdream.net) has joined #ceph
[0:49] * brambles (brambles@79.133.200.49) Quit (Quit: leaving)
[0:49] * brambles (brambles@79.133.200.49) has joined #ceph
[0:53] <iggy> damnit gregaf... how did you get them to understand that with 1 line and it took me over an hour last night, and I still don't think that guy understood it
[0:54] * sagelap (~sage@ace.ops.newdream.net) Quit (Quit: Leaving.)
[1:06] <Tv_> ubuntu@vercoi02:~$ ip -o li |wc -l
[1:06] <Tv_> 36
[1:06] <Tv_> mwahahaha
[1:16] * sjustlaptop (~sam@aon.hq.newdream.net) Quit (Read error: Connection reset by peer)
[1:17] * sjustlaptop (~sam@aon.hq.newdream.net) has joined #ceph
[1:18] <nhm> gregaf: so on the other aging cluster, it looks like something blew up all 3 mons at the same time.
[1:18] <Tv_> ruh-roh network trouble on sepia
[1:19] <nhm> gregaf: when I say blew up, I mean took the kernel down too.
[1:22] <gregaf> I blame TV and/or the network trouble
[1:22] <nhm> gregaf: oooh, looks like it's the same problem as earlier, and this was with debug 10 on the mon
[1:23] <nhm> it took all 3 mons and a client down with it.
[1:23] <gregaf> I don't see how it could take down the kernel?
[1:23] <gregaf> ah, a kernel client, you mean?
[1:23] <nhm> gregaf: all 3 mon nodes and a client node were all down.
[1:25] <gregaf> okay
[1:25] <nhm> gregaf: if you are interested, take a look at the mon log on plana82 at around: 2012-06-20 14:55:09
[1:27] <nhm> hrm, happened after the mon shutdown if I'm reading that right.
[1:28] <gregaf> I can't get in...
[1:28] <gregaf> Tv_: what kind of network trouble?
[1:28] <nhm> gregaf: I'll pastie
[1:28] <Tv_> gregaf: something got messed up in vercoi networking, and we lost dns temporarily
[1:28] <gregaf> k, just checking
[1:29] <gregaf> means I can't get in right now though, nhm ?????we'll have to do this through you
[1:29] <gregaf> (or just save it somewhere?)
[1:30] <Tv_> oh the vm dropped off the bridge
[1:30] <Tv_> hrmmph
[1:31] <nhm> gregaf: http://www.pastie.org/4151298
[1:33] <Tv_> and we should be back
[1:35] <gregaf> okay, so it looks like it???got a probe from a monitor and then tried to open a new pipe to it for the reply?!?
[1:36] * BManojlovic (~steki@212.200.241.3) Quit (Quit: Ja odoh a vi sta 'ocete...)
[1:38] <nhm> gregaf: I don't know enough about the monitors to even begin to know. Is it getting probed or doing the probing?
[1:38] <gregaf> nhm: it's sort of a mutual probe
[1:39] <gregaf> with a double-ended instrument
[1:39] <nhm> how very consensual.
[1:39] <gregaf> only this monitor is breaking the rules and trying to introduce a new instrument, and that causes everything to fall apart
[1:39] * Tv_ (~tv@aon.hq.newdream.net) Quit (Quit: Tv_)
[1:42] <nhm> gregaf: So is this the same thing as before?
[1:42] <gregaf> possibly?
[1:42] <nhm> Add to the ticket?
[1:42] <gregaf> yes please
[1:44] <gregaf> that should be very helpful in figuring out what's going on, I'm just trying to pretend I'm a sysadmin with chef right now so I'm trying to avoid getting distracted by more interesting problems :)
[1:46] <nhm> lol
[2:00] <gregaf> nhm: wrong bug for that update?
[2:06] <nhm> gregaf: oh crap, sorry about that.
[2:06] <nhm> any easy way to fix?
[2:07] * yehudasa_ (~yehudasa@aon.hq.newdream.net) Quit (Ping timeout: 480 seconds)
[2:08] <gregaf> copy it over to the right bug, and edit your post to say "wrong place"?
[2:08] <gregaf> I don't seem to have a "delete comment" button or anything
[2:08] <nhm> yeah, I don't have a delete comment button, and I was never able to figure out how to edit comments.
[2:09] <nhm> Maybe a permission issue or something.
[2:10] <nhm> alright, put the comment on the right bug. Gotta run now. Will try to figure out how to fix the other one later (feel free to do so if you can)
[2:14] * The_Bishop (~bishop@p5DC11956.dip.t-dialin.net) Quit (Quit: Wer zum Teufel ist dieser Peer? Wenn ich den erwische dann werde ich ihm mal die Verbindung resetten!)
[2:57] * Solver_ is now known as Solver
[3:17] * sjustlaptop (~sam@aon.hq.newdream.net) Quit (Ping timeout: 480 seconds)
[3:43] <renzhi> Hi, if I only use native rados, I don't really need mds, right?
[3:45] <renzhi> i.e. I don't mount the file system, but just access the cluster through librados
[3:48] <dmick> renzhi: that's correct
[3:48] <dmick> mds is only for cephfs
[3:50] <renzhi> thx, I'm trying to do hardware cost estimation :)
[3:51] <renzhi> The wiki said, you need 1-2GB of RAM for each osd, does that depend on the size of the underlying storage volume of each osd?
[3:54] <renzhi> also, does it depend on the weight of the osd, as defined in ceph.conf?
[4:06] * sjustlaptop (~sam@24-205-39-1.dhcp.gldl.ca.charter.com) has joined #ceph
[4:16] <dmick> not sure on that one, sorry
[4:16] * Ryan_Lane1 (~Adium@dslb-088-075-181-007.pools.arcor-ip.net) has joined #ceph
[4:23] * Ryan_Lane (~Adium@dslb-178-008-042-229.pools.arcor-ip.net) Quit (Ping timeout: 480 seconds)
[4:24] <renzhi> ok :)
[4:46] * OutBackDingo (~quassel@rrcs-71-43-84-222.se.biz.rr.com) has joined #ceph
[5:25] * chutzpah (~chutz@216.174.109.254) Quit (Quit: Leaving)
[5:32] <nhm> ugh, the ssh keys changed on the aging clusters again. This is not cool.
[5:42] <nhm> hrm, it wasn't the ssh key, just the known_hosts file.
[5:43] <nhm> guess I'll switch these tests to just run as another user.
[5:45] <nhm> gah, authorized_keys, not known_hosts. That's what I get for trying to think after jackhammering concrete.
[5:53] * adjohn (~adjohn@50-0-133-101.dsl.static.sonic.net) Quit (Quit: adjohn)
[5:58] <dmick> oh, crap, I hope I didn't do this to you again
[5:58] * adjohn (~adjohn@50-0-133-101.dsl.static.sonic.net) has joined #ceph
[5:58] <dmick> I probably did; I noticed that at least one of them was screwed with multiple keys per line, and refreshed all the ones that were
[5:58] <dmick> but of course refreshed them from the master list
[5:59] <dmick> we could just put your keys in the master list, I suppose
[6:31] * cattelan (~cattelan@2001:4978:267:0:21c:c0ff:febf:814b) Quit (Ping timeout: 480 seconds)
[6:38] * Ryan_Lane1 (~Adium@dslb-088-075-181-007.pools.arcor-ip.net) Quit (Ping timeout: 480 seconds)
[6:40] * cattelan (~cattelan@2001:4978:267:0:21c:c0ff:febf:814b) has joined #ceph
[6:45] * dmick (~dmick@aon.hq.newdream.net) Quit (Quit: Leaving.)
[6:50] * cattelan is now known as cattelan_away
[7:03] * adjohn (~adjohn@50-0-133-101.dsl.static.sonic.net) Quit (Quit: adjohn)
[7:25] * adjohn (~adjohn@50-0-133-101.dsl.static.sonic.net) has joined #ceph
[7:28] * adjohn (~adjohn@50-0-133-101.dsl.static.sonic.net) Quit ()
[7:38] * adjohn (~adjohn@50-0-133-101.dsl.static.sonic.net) has joined #ceph
[7:39] * adjohn (~adjohn@50-0-133-101.dsl.static.sonic.net) Quit ()
[7:58] * loicd (~loic@2a01:e35:2eba:db10:120b:a9ff:feb7:cce0) has joined #ceph
[8:06] * loicd (~loic@2a01:e35:2eba:db10:120b:a9ff:feb7:cce0) Quit (Quit: Leaving.)
[8:12] * sjustlaptop (~sam@24-205-39-1.dhcp.gldl.ca.charter.com) Quit (Ping timeout: 480 seconds)
[8:21] * OutBackDingo (~quassel@rrcs-71-43-84-222.se.biz.rr.com) Quit (Remote host closed the connection)
[8:23] * OutBackDingo (~quassel@rrcs-71-43-84-222.se.biz.rr.com) has joined #ceph
[8:31] * adjohn (~adjohn@50-0-133-101.dsl.static.sonic.net) has joined #ceph
[8:42] * renzhi (~renzhi@66.197.225.53) Quit (Read error: Connection reset by peer)
[8:42] * renzhi (~renzhi@66.197.225.53) has joined #ceph
[9:03] * adjohn (~adjohn@50-0-133-101.dsl.static.sonic.net) Quit (Quit: adjohn)
[9:09] * BManojlovic (~steki@91.195.39.5) has joined #ceph
[9:11] * loicd (~loic@83.167.43.235) has joined #ceph
[9:25] * adjohn (~adjohn@50-0-133-101.dsl.static.sonic.net) has joined #ceph
[9:26] * adjohn (~adjohn@50-0-133-101.dsl.static.sonic.net) Quit ()
[9:38] * verwilst (~verwilst@d5152FEFB.static.telenet.be) has joined #ceph
[9:40] * mtk (~mtk@ool-44c35bb4.dyn.optonline.net) Quit (Ping timeout: 480 seconds)
[9:51] * mtk (~mtk@ool-44c35bb4.dyn.optonline.net) has joined #ceph
[10:00] * s[X]_ (~sX]@eth589.qld.adsl.internode.on.net) Quit (Remote host closed the connection)
[10:00] * lotreck (~Adium@c-75-75-33-53.hsd1.va.comcast.net) Quit (Quit: Leaving.)
[10:28] * s[X]_ (~sX]@ppp59-167-157-96.static.internode.on.net) has joined #ceph
[11:58] * s[X]_ (~sX]@ppp59-167-157-96.static.internode.on.net) Quit (Remote host closed the connection)
[12:11] * s[X]_ (~sX]@ppp59-167-157-96.static.internode.on.net) has joined #ceph
[12:21] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has joined #ceph
[12:56] * nhorman (~nhorman@2001:470:8:a08:7aac:c0ff:fec2:933b) has joined #ceph
[13:32] <nhm> dmick: Yeah, the other option (that I should have probably started doing long ago) is to create and run these tests under their own account.
[13:35] <jluis> nhm, I am curious. Jackhammering concrete?
[13:41] <nhm> jluis: Fixing my house. ;P
[13:42] <nhm> jluis: We had to remove some concrete stairs to get at a rimjoist that is rotting out. Got rid of the stairs, and almost have the new joist in. Just need to rip the wood, get it in place, and anchor it now.
[13:58] <jluis> damn
[13:59] <jluis> sounds a lot of fun to just fix some stuff
[13:59] <jluis> I should start fixing stuff around here too. With a jackhammer.
[13:59] <nhm> jluis: yeah, jackhammering stairs was not on my list of summer projects I was intending to do.
[14:00] <nhm> jluis: lol, it's actually easier than you'd think. Sounds like the newer jackhammers vibrate a lot less than the old ones.
[14:00] <nhm> Most of the work is just lifting it up to reposition it.
[14:02] <jluis> :D
[14:02] <jluis> so, now that you had a project involving a jackhammer, what are you going to do to have an excuse to use a flame thrower?
[14:03] <nhm> jluis: well, my son's birthday party is coming up. I suppose we could barbeque...
[14:03] <jluis> already sounds like the best birthday party ever :p
[14:09] <nhm> ugh, these aging tests never want to run more than a couple of hours. :(
[14:10] <jluis> damn... I miss LA's weather
[14:11] <jluis> it was hot, but not that humid
[14:11] <nhm> jluis: yeah, it's been really humid here too.
[14:11] <nhm> Not super hot, but you still feel miserable.
[14:11] <jluis> yeah, it's pretty damn humid here
[14:12] <jluis> and 32C
[14:12] <jluis> the weather people say we may reach 40C today or tomorrow
[14:13] <jluis> (that would be 89.6F according to google)
[14:14] <nhm> Yeah, it was about 29c here yesterday, but really humid. 40C and humid would be awful. :(
[14:14] <nhm> that's about the hottest it gets here.
[14:15] <nhm> And only rarely. Usually our summer highs are more like 35-37c.
[14:15] <liiwi> humidity does have it's sides, less goo getting into nose and eyes
[14:16] <jluis> yet, more goo sticking your clothes to your body :p
[14:48] * Ryan_Lane (~Adium@dslb-088-075-181-007.pools.arcor-ip.net) has joined #ceph
[15:04] * plastics (~plastics@c-69-138-42-222.hsd1.tn.comcast.net) has joined #ceph
[15:09] * lofejndif (~lsqavnbok@19NAAAECF.tor-irc.dnsbl.oftc.net) has joined #ceph
[15:30] * cattelan_away is now known as cattelan
[15:48] * plastics (~plastics@c-69-138-42-222.hsd1.tn.comcast.net) Quit (Ping timeout: 480 seconds)
[16:09] * plastics (~plastics@c-69-138-42-222.hsd1.tn.comcast.net) has joined #ceph
[16:23] * lofejndif (~lsqavnbok@19NAAAECF.tor-irc.dnsbl.oftc.net) Quit (Quit: gone)
[17:12] * s[X]_ (~sX]@ppp59-167-157-96.static.internode.on.net) Quit (Remote host closed the connection)
[17:14] * sjustlaptop (~sam@24-205-39-1.dhcp.gldl.ca.charter.com) has joined #ceph
[17:15] * widodh_ (~widodh@minotaur.apache.org) Quit (Read error: Connection reset by peer)
[17:15] * widodh (~widodh@minotaur.apache.org) has joined #ceph
[17:22] * sjustlaptop (~sam@24-205-39-1.dhcp.gldl.ca.charter.com) Quit (Ping timeout: 480 seconds)
[17:23] * plastics (~plastics@c-69-138-42-222.hsd1.tn.comcast.net) Quit (Ping timeout: 480 seconds)
[17:24] * verwilst (~verwilst@d5152FEFB.static.telenet.be) Quit (Quit: Ex-Chat)
[17:30] * sagelap (~sage@64.sub-166-250-65.myvzw.com) has joined #ceph
[17:30] <- *sagelap* .1
[17:34] * BManojlovic (~steki@91.195.39.5) Quit (Quit: Ja odoh a vi sta 'ocete...)
[17:38] * sagelap1 (~sage@83.sub-166-250-73.myvzw.com) has joined #ceph
[17:38] * sagelap (~sage@64.sub-166-250-65.myvzw.com) Quit (Ping timeout: 480 seconds)
[17:40] * bchrisman (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[17:42] * Tv_ (~tv@aon.hq.newdream.net) has joined #ceph
[17:46] * sagelap1 (~sage@83.sub-166-250-73.myvzw.com) Quit (Ping timeout: 480 seconds)
[17:56] * sagelap (~sage@64.sub-166-250-65.myvzw.com) has joined #ceph
[17:58] * lofejndif (~lsqavnbok@83TAAGWUH.tor-irc.dnsbl.oftc.net) has joined #ceph
[18:06] * sagelap (~sage@64.sub-166-250-65.myvzw.com) Quit (Ping timeout: 480 seconds)
[18:14] <nhm> via teuthology, what would be the best way to figure out which devices were selected for ceph. Just parse the generated ceph.conf file?
[18:18] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) Quit (Ping timeout: 480 seconds)
[18:22] * elder (~elder@aon.hq.newdream.net) has joined #ceph
[18:22] * elder (~elder@aon.hq.newdream.net) Quit (Remote host closed the connection)
[18:25] * elder (~elder@aon.hq.newdream.net) has joined #ceph
[18:27] * aliguori (~anthony@cpe-70-123-145-39.austin.res.rr.com) Quit (Quit: Ex-Chat)
[18:32] * nhorman (~nhorman@2001:470:8:a08:7aac:c0ff:fec2:933b) Quit (Ping timeout: 480 seconds)
[18:33] * sagelap (~sage@aon.hq.newdream.net) has joined #ceph
[18:37] * nhorman (~nhorman@2001:470:8:a08:7aac:c0ff:fec2:933b) has joined #ceph
[18:38] * ninkotech (~duplo@89.177.137.231) Quit (Remote host closed the connection)
[18:43] * yehudasa_ (~yehudasa@aon.hq.newdream.net) has joined #ceph
[18:43] <gregaf> nhm: if you're asking about which nodes got assigned what roles, yes ??? there's not currently an output of that anywhere in the log; it's annoying
[18:47] <nhm> gregaf: that would be useful too. Right now I'm trying to figure out which devices got assigned to what, IE, did /dev/sdd get used and for what?
[18:48] <nhm> and yeah, I guess I would also need to know which node too.
[18:48] <gregaf> oh, I see
[18:48] <gregaf> I think that involves parsing the config file, yes
[18:48] <gregaf> Tv_ or Josh (not on right now?) might have better ideas, though
[18:49] * sagelap (~sage@aon.hq.newdream.net) Quit (Quit: Leaving.)
[18:50] * bchrisman (~Adium@108.60.121.114) has joined #ceph
[18:52] * plastics (~plastics@c-69-138-42-222.hsd1.tn.comcast.net) has joined #ceph
[18:54] * widodh_ (~widodh@minotaur.apache.org) has joined #ceph
[18:54] * joshd (~joshd@aon.hq.newdream.net) has joined #ceph
[18:58] * MarkDude (~MT@c-71-198-138-155.hsd1.ca.comcast.net) Quit (Quit: Leaving)
[18:59] * MarkDude (~MT@c-71-198-138-155.hsd1.ca.comcast.net) has joined #ceph
[18:59] * widodh (~widodh@minotaur.apache.org) Quit (Ping timeout: 480 seconds)
[19:06] <Tv_> all 4-letter identifiers are equal, right? *headdesk*
[19:07] <gregaf> ???.what?
[19:07] <Tv_> assert "user" == "meta"
[19:07] <Tv_> that took way too long to see
[19:07] * loicd (~loic@83.167.43.235) Quit (Quit: Leaving.)
[19:08] <gregaf> haha
[19:08] <Tv_> i kept passing downburst a --user-data=foo.meta.yaml argument happily for half an hour
[19:09] * sagelap (~sage@aon.hq.newdream.net) has joined #ceph
[19:13] <nhm> hrm, guess goto meeting doesn't support linux
[19:14] <elder> Use your phone.
[19:15] <nhm> I'm trying osx on my laptop for now
[19:15] <nhm> will do the phone as backup
[19:18] * lofejndif (~lsqavnbok@83TAAGWUH.tor-irc.dnsbl.oftc.net) Quit (Quit: gone)
[19:24] * OutBackDingo (~quassel@rrcs-71-43-84-222.se.biz.rr.com) has left #ceph
[19:25] * chutzpah (~chutz@216.174.109.254) has joined #ceph
[19:26] * vikasap (~vikasap@64.209.89.103) Quit (Quit: leaving)
[19:34] * BManojlovic (~steki@212.200.241.3) has joined #ceph
[19:37] * loicd (~loic@magenta.dachary.org) has joined #ceph
[19:37] * siren (~siren@19NAAAEQ0.tor-irc.dnsbl.oftc.net) has joined #ceph
[19:46] * aliguori (~anthony@32.97.110.59) has joined #ceph
[19:47] * dmick (~dmick@aon.hq.newdream.net) has joined #ceph
[19:49] * adjohn (~adjohn@50-0-133-101.dsl.static.sonic.net) has joined #ceph
[20:17] * bakher (~siren@83TAAGW0Q.tor-irc.dnsbl.oftc.net) has joined #ceph
[20:21] * siren (~siren@19NAAAEQ0.tor-irc.dnsbl.oftc.net) Quit (Remote host closed the connection)
[20:40] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[20:40] * loicd (~loic@magenta.dachary.org) has joined #ceph
[20:56] <bakher> fuck Justin Bieber | fuck Justin Bieber | fuck Justin Bieber | fuck Justin Bieber | fuck Justin Bieber | fuck Justin Bieber | fuck Justin Bieber | fuck Justin Bieber | fuck Justin Bieber | fuck Justin Bieber | fuck Justin Bieber | fuck Justin Bieber | fuck Justin Bieber | fuck Justin Bieber | fuck Justin Bieber | fuck Justin Bieber | fuck Justin Bieber | fuck Justin Bieber | fuck Justin Bieber
[20:56] <bakher> | fuck Justin Bieber | fuck Justin Bieber | fuck Justin Bieber | fuck Justin Bieber | fuck Justin Bieber | fuck Justin Bieber | fuck Justin Bieber | fuck Justin Bieber | fuck Justin Bieber | fuck Justin Bieber | fuck Justin Bieber | fuck Justin Bieber |
[21:00] * bakher (~siren@83TAAGW0Q.tor-irc.dnsbl.oftc.net) Quit (Killed (tjfontaine (No reason)))
[21:03] * mgalkiewicz (~mgalkiewi@toya.hederanetworks.net) has joined #ceph
[21:06] <mgalkiewicz> hi guys I have a problem with shutting down VM which has mounted rbd volume https://gist.github.com/2998077
[21:06] <mgalkiewicz> shutdown just hangs and it looks like it waits for rbd
[21:06] <mgalkiewicz> I can reproduce the problem
[21:20] * ninkotech (~duplo@89.177.137.231) has joined #ceph
[21:33] * lofejndif (~lsqavnbok@9YYAAHL4E.tor-irc.dnsbl.oftc.net) has joined #ceph
[21:40] * mgalkiewicz_ (~mgalkiewi@toya.hederanetworks.net) has joined #ceph
[21:55] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has joined #ceph
[21:58] <joshd> mgalkiewicz_: is there anything wrong with osd.1?
[21:59] <joshd> mgalkiewicz_: if you have the admin socket enabled for the osd, does 'ceph --admin-daemon /path/to/ceph-osd.1.asok dump_ops_in_flight' show anything from that rbd volume?
[22:00] * plastics (~plastics@c-69-138-42-222.hsd1.tn.comcast.net) Quit (Ping timeout: 480 seconds)
[22:02] <joshd> mgalkiewicz: it may be http://tracker.newdream.net/issues/2600
[22:20] * The_Bishop (~bishop@2a01:198:2ee:0:913:2288:fd08:3bae) has joined #ceph
[22:34] * nhorman (~nhorman@2001:470:8:a08:7aac:c0ff:fec2:933b) Quit (Quit: Leaving)
[22:41] <mgalkiewicz_> joshd: not sure how to run this ceph --admin -daemon
[22:43] <joshd> mgalkiewicz_: do you have a /var/run/ceph/ceph-osd.1.asok file?
[22:44] <joshd> mgalkiewicz_: that's the default path for the admin socket, but it could be changed, e.g. 'admin socket = /path/to/socket/$name.sock' in ceph.conf
[22:45] <joshd> mgalkiewicz_: you'd need to run that command on the machine where osd.1 is running
[22:45] <mgalkiewicz_> ok I got asok file
[22:47] <mgalkiewicz_> joshd: https://gist.github.com/2998860
[22:50] <joshd> ok, so that's no requests in flight
[22:50] * aliguori (~anthony@32.97.110.59) Quit (Remote host closed the connection)
[22:51] <joshd> which version of the kernel are you using?
[22:51] <mgalkiewicz_> 3.2.20
[22:51] <mgalkiewicz_> 3.2.19
[22:51] <mgalkiewicz_> the second one
[22:57] <joshd> mgalkiewicz_: if you have debugging enabled, could you turn it on for libceph and reproduce? i.e. echo 'module libceph +p' > /sys/kernel/debug/dynamic_debug/control
[22:57] <mgalkiewicz_> on client side?
[22:58] <mgalkiewicz_> please be more specific
[22:58] <joshd> yeah, on the client side
[22:58] <mgalkiewicz_> ok how to enable debugging? just in ceph.conf?
[22:59] <joshd> it's compiled into the kernel for the kernel client
[22:59] <joshd> this enables it and should log to syslog: echo 'module libceph +p' > /sys/kernel/debug/dynamic_debug/control
[23:02] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has left #ceph
[23:02] * adjohn (~adjohn@50-0-133-101.dsl.static.sonic.net) Quit (Quit: adjohn)
[23:06] <mgalkiewicz_> joshd: checking
[23:11] <mgalkiewicz_> -bash: /sys/kernel/debug/dynamic_debug/control: No such file or directory
[23:16] <joshd> mgalkiewicz_: does 'mount -t debugfs none /sys/kernel/debug' work?
[23:21] <mgalkiewicz_> yes
[23:21] <mgalkiewicz_> but previous command still doesnt
[23:22] * aliguori (~anthony@cpe-70-123-145-39.austin.res.rr.com) has joined #ceph
[23:23] * lofejndif (~lsqavnbok@9YYAAHL4E.tor-irc.dnsbl.oftc.net) Quit (Ping timeout: 480 seconds)
[23:24] * s[X]_ (~sX]@ppp59-167-157-96.static.internode.on.net) has joined #ceph
[23:26] <dmick> you might need a kernel option for that
[23:26] <dmick> CONFIG_DYNAMIC_DEBUG http://lwn.net/Articles/434833/
[23:26] <dmick> not sure whether to expect that in, say, precise. looking
[23:29] <dmick> it's not set in my default config in /boot
[23:29] * s[X]_ (~sX]@ppp59-167-157-96.static.internode.on.net) Quit (Remote host closed the connection)
[23:29] <mgalkiewicz_> my neither
[23:34] * lofejndif (~lsqavnbok@04ZAAD4UI.tor-irc.dnsbl.oftc.net) has joined #ceph
[23:34] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[23:34] * loicd (~loic@2a01:e35:2eba:db10:120b:a9ff:feb7:cce0) has joined #ceph
[23:40] <Tv_> dmick: yeah that's one of the things we set explicitly for our own kernel builds
[23:46] <dmick> according to the lwn.net article, that stuff should still be in a printk somewhere
[23:46] <dmick> headers seem to bear that out
[23:47] <dmick> assuming, I guess, that DEBUG is defined
[23:49] <dmick> ah, but the ceph header is a wrinkle
[23:58] <mgalkiewicz_> ok so I see that without this kernel option I cant provide you enough log. I will try to recompile and let you know.
[23:59] * adjohn (~adjohn@50-0-133-101.dsl.static.sonic.net) has joined #ceph

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.