#ceph IRC Log


IRC Log for 2012-08-14

Timestamps are in GMT/BST.

[0:04] * tnt (~tnt@17.127-67-87.adsl-dyn.isp.belgacom.be) Quit (Ping timeout: 480 seconds)
[0:12] * lofejndif (~lsqavnbok@28IAAGR6H.tor-irc.dnsbl.oftc.net) has joined #ceph
[0:16] * adjohn (~adjohn@ has joined #ceph
[0:21] * maelfius (~Adium@ has joined #ceph
[0:21] * MarkN (~nathan@ has joined #ceph
[0:22] * MarkN (~nathan@ has left #ceph
[0:22] * Leseb (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) Quit (Quit: Leseb)
[0:22] <maelfius> so I created an RBD image and have it mounted. The image is 5000MB in size
[0:22] <maelfius> however ceph -w is showing 2561MB used
[0:22] <maelfius> is this due to some lazy allocation?
[0:22] <joshd> they're thin provisioned
[0:23] <maelfius> joshd: cool, just confirming
[0:23] <lurbs> There's no way to reclaim space from them once it's been used, is there?
[0:23] <joshd> they support trim/discard in qemu
[0:24] <joshd> it's a bit limited though - virtio doesn't support discard, you need to use scsci or ide
[0:24] <iggy> or virtio-scsi
[0:26] <Tv_> there's no reason why somebody wouldn't add that to virtio too, though; doesn't invalidate the approach at all
[0:27] <joshd> yeah, Paolo Bonzini has said he wants to do that, but it may take some time
[0:27] <iggy> nobody seems to have done much with virtio-blk features lately... everybody seems busy with basically gutting and rewriting qemu's core
[0:29] * steki-BLAH (~steki@ has joined #ceph
[0:34] * lofejndif (~lsqavnbok@28IAAGR6H.tor-irc.dnsbl.oftc.net) Quit (Remote host closed the connection)
[0:35] * lofejndif (~lsqavnbok@28IAAGR63.tor-irc.dnsbl.oftc.net) has joined #ceph
[0:36] * BManojlovic (~steki@ Quit (Ping timeout: 480 seconds)
[1:15] * steki-BLAH (~steki@ Quit (Quit: Ja odoh a vi sta 'ocete...)
[1:18] * yoshi (~yoshi@p22043-ipngn1701marunouchi.tokyo.ocn.ne.jp) has joined #ceph
[1:40] * bchrisman (~Adium@ Quit (Quit: Leaving.)
[1:50] * Tv_ (~tv@2607:f298:a:607:60e1:881b:ec67:50fe) Quit (Quit: Tv_)
[1:57] * lofejndif (~lsqavnbok@28IAAGR63.tor-irc.dnsbl.oftc.net) Quit (Quit: gone)
[2:34] * Cube (~Adium@ Quit (Quit: Leaving.)
[2:56] * joshd (~joshd@2607:f298:a:607:221:70ff:fe33:3fe3) Quit (Quit: Leaving.)
[2:59] * Cube (~Adium@cpe-76-95-223-199.socal.res.rr.com) has joined #ceph
[3:02] * The_Bishop (~bishop@p4FCDF575.dip.t-dialin.net) Quit (Quit: Wer zum Teufel ist dieser Peer? Wenn ich den erwische dann werde ich ihm mal die Verbindung resetten!)
[3:04] * bchrisman (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) has joined #ceph
[3:13] * adjohn (~adjohn@ Quit (Quit: adjohn)
[3:53] * chutzpah (~chutz@ Quit (Quit: Leaving)
[4:31] * glowell (~Adium@c-98-210-226-131.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[4:37] * maelfius (~Adium@ Quit (Quit: Leaving.)
[4:43] * glowell (~Adium@c-98-210-226-131.hsd1.ca.comcast.net) has joined #ceph
[5:40] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[5:41] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[5:58] * Qu310 (Q@qten.qnet.net.au) has joined #ceph
[5:58] * Qten (Q@qten.qnet.net.au) Quit (Read error: Connection reset by peer)
[6:24] <lxo> yay, I've just debugged the blocked access to hard links that I reported here a while ago. the mds fails to respond to itself the anchortable requests
[6:25] <lxo> patch is now running on 0.48argonaut, will forward-port it to 0.50 and post it
[6:27] * maelfius (~Adium@pool-71-160-33-115.lsanca.fios.verizon.net) has joined #ceph
[6:53] * Ryan_Lane1 (~Adium@c-67-160-217-184.hsd1.ca.comcast.net) has joined #ceph
[6:56] * Ryan_Lane (~Adium@c-67-160-217-184.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[6:59] * Ryan_Lane1 (~Adium@c-67-160-217-184.hsd1.ca.comcast.net) Quit (Read error: Operation timed out)
[7:09] * Ryan_Lane (~Adium@c-67-160-217-184.hsd1.ca.comcast.net) has joined #ceph
[7:23] * deepsa_ (~deepsa@ has joined #ceph
[7:24] * deepsa (~deepsa@ Quit (Ping timeout: 480 seconds)
[7:24] * deepsa_ is now known as deepsa
[7:30] * joshd (~jdurgin@2602:306:c5db:310:1e6f:65ff:feaa:beb7) has joined #ceph
[7:40] * Tv_ (~tv@ has joined #ceph
[7:53] * tnt (~tnt@17.127-67-87.adsl-dyn.isp.belgacom.be) has joined #ceph
[8:02] * liiwi (liiwi@idle.fi) Quit (Quit: leaving)
[8:02] * liiwi (liiwi@idle.fi) has joined #ceph
[8:14] * EmilienM (~EmilienM@ has joined #ceph
[8:25] * Leseb (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) has joined #ceph
[8:26] * maelfius (~Adium@pool-71-160-33-115.lsanca.fios.verizon.net) Quit (Quit: Leaving.)
[8:29] * Leseb (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) Quit ()
[8:42] * Anticime1 is now known as Anticimex
[8:50] <NaioN> sjust: thx, will try the patch and report back!
[9:11] * Leseb (~Leseb@2001:980:759b:1:fc21:d9bb:42b3:d171) has joined #ceph
[9:12] * Leseb_ (~Leseb@ has joined #ceph
[9:17] * tnt (~tnt@17.127-67-87.adsl-dyn.isp.belgacom.be) Quit (Ping timeout: 480 seconds)
[9:19] * Leseb (~Leseb@2001:980:759b:1:fc21:d9bb:42b3:d171) Quit (Ping timeout: 480 seconds)
[9:19] * Leseb_ is now known as Leseb
[9:26] * verwilst (~verwilst@d5152FEFB.static.telenet.be) has joined #ceph
[9:33] * tnt (~tnt@212-166-48-236.win.be) has joined #ceph
[9:36] * BManojlovic (~steki@ has joined #ceph
[9:57] * fghaas (~florian@91-119-225-146.dynamic.xdsl-line.inode.at) has joined #ceph
[10:04] * fghaas (~florian@91-119-225-146.dynamic.xdsl-line.inode.at) Quit (Quit: Leaving.)
[10:18] * loicd1 is now known as loicd
[10:19] * Ryan_Lane (~Adium@c-67-160-217-184.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[11:14] * kermin (ca037809@ircip2.mibbit.com) has joined #ceph
[11:16] <kermin> hi any1 der?
[11:21] * yoshi (~yoshi@p22043-ipngn1701marunouchi.tokyo.ocn.ne.jp) Quit (Remote host closed the connection)
[11:23] * Ryan_Lane (~Adium@c-67-160-217-184.hsd1.ca.comcast.net) has joined #ceph
[11:28] <kermin> ??
[11:29] <liiwi> I think most peeps are from west coast us, so they haven't woken up yet
[11:30] <kermin> oww..
[11:31] <kermin> can u help me out with ERROR: error creating empty object store in /ceph_data/osd.0: (2) No such file or directory
[11:31] * Ryan_Lane (~Adium@c-67-160-217-184.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[11:31] <kermin> i ws executing " mkcephfs -a -c ceph.conf -k ceph.keyring"
[11:40] <liiwi> hmm, check where the object store is
[11:47] <joao> the osd.0 directory is missing
[11:48] <joao> I'm not sure if mkcephfs creates de osd directories
[11:49] <tnt> no, you need to create the directory yourself.
[12:02] <NaioN> indeed
[12:03] <NaioN> the directory has to exist
[12:03] <NaioN> and normally you also need to mount a fs under /ceph_data/osd.0
[12:05] <kermin> i tried it sir
[12:05] <kermin> mannual i created d dir
[12:05] <kermin> bt still same error is fired
[12:07] <kermin> also i executed "mount -o remount,user_xattr /"
[12:08] <kermin> it works fine
[12:08] <kermin> problem is with "sudo mkcephfs -a -c /etc/ceph/ceph.conf -k ceph.keyring" only
[13:14] <kermin> @NaioN - can you please suggest me the command to mount a fs under /ceph_data/osd.0 ?
[13:14] <cephalobot`> kermin: Error: "NaioN" is not a valid command.
[13:14] <kermin> can you please suggest me the command to mount a fs under /ceph_data/osd.0 ?
[13:14] * nhorman (~nhorman@hmsreliant.think-freely.org) has joined #ceph
[13:23] <joao> kermin, mount /dev/something /ceph_data/osd.0 ?
[13:25] * kermin (ca037809@ircip2.mibbit.com) Quit (Quit: http://www.mibbit.com ajax IRC Client)
[13:29] * kermin (ca037809@ircip3.mibbit.com) has joined #ceph
[14:15] <kermin> hi !
[14:15] <kermin> any1 der?
[14:18] * ninkotech (~duplo@ Quit (Quit: Konversation terminated!)
[14:21] * ssedov (stas@ssh.deglitch.com) Quit (Read error: Connection reset by peer)
[14:21] * stass (stas@ssh.deglitch.com) has joined #ceph
[14:34] * ninkotech (~duplo@ has joined #ceph
[15:36] * dspano (~dspano@rrcs-24-103-221-202.nys.biz.rr.com) has joined #ceph
[15:43] * kermin (ca037809@ircip3.mibbit.com) Quit (Quit: http://www.mibbit.com ajax IRC Client)
[16:34] <tjpatter> Any helpful hints as to how I can easily get the rbd kernel module on CentOS 6.x???
[16:35] <pmjdebruijn> you'd probably need to replace the whole kernel
[16:36] <pmjdebruijn> which I guess is the main point of running CentOS in the first place
[16:36] <pmjdebruijn> tjpatter: stick around, maybe someone else knows better
[16:36] <tjpatter> Hmm...
[16:36] * pmjdebruijn runs rbd on ubuntu
[16:36] <pmjdebruijn> but we roll our own kernels anyhow
[16:37] <tjpatter> I have it running on Ubuntu. Easy there. Wondering if there is a loadable module for CentOS. It would be very painful for us to have to roll a custom kernel in our environment.
[16:37] <pmjdebruijn> like I said, stick around, maybe someone else knows better
[16:37] <tnt> pmjdebruijn: which one are you using ? I just tried 3.6-rc1 to get the latest ceph stuff and network just collapse on it ...
[16:37] <pmjdebruijn> we run our test clutser on 3.5.1 now
[16:39] <pmjdebruijn> we generally try to stay with the -stable tree's when possible
[16:44] <tnt> I tried 3.5.1 now but it's missing a shitload of ceph patches :(
[16:44] <pmjdebruijn> cephfs? or rbd?
[16:44] <tnt> also for some reason loading the netconsole module crashes it.
[16:44] <pmjdebruijn> we don't use cephfs for that matter
[16:45] <pmjdebruijn> or libceph?
[16:45] <tnt> libceph sorry.
[16:45] <pmjdebruijn> ah
[16:45] <pmjdebruijn> so that's matters for both
[16:45] <pmjdebruijn> ideally we would have liked to stick with 3.2.0 as that will be getting long term maintenance
[16:47] <tnt> yup me too but unfortunately I need the latest rbd stuff and I also need some network stuff that appeared in 3.3 ...
[16:47] <tnt> I'll probably start backporting all the ceph patches to 3.5.1 now.
[16:48] <pmjdebruijn> I thought there was some effort to maintain 3.4.x with newer ceph stuff for some time
[16:55] * NashTrash (~Adium@mobile-166-147-119-148.mycingular.net) has joined #ceph
[16:55] <NashTrash> Hello Ceph'ers
[16:56] <NashTrash> Tv_: You around?
[16:59] <NashTrash> Is it OK to set mod addr to Or, is it possible to have multiple public addresses for the various services?
[17:01] * laks (~laks@ has joined #ceph
[17:12] * BManojlovic (~steki@ Quit (Quit: Ja odoh a vi sta 'ocete...)
[17:23] * laks (~laks@ Quit (Quit: leaving)
[17:25] <tjpatter> To add to my previous message here??? I would also be okay with a way to rebuild qemu-kvm on CentOS to include support for rbd directly from qemu-img
[17:28] * verwilst (~verwilst@d5152FEFB.static.telenet.be) Quit (Quit: Ex-Chat)
[17:33] * vhasi (martin@vha.si) Quit (Read error: Operation timed out)
[17:33] * vhasi (vhasi@vha.si) has joined #ceph
[17:34] * bchrisman (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[17:59] * tnt (~tnt@212-166-48-236.win.be) Quit (Ping timeout: 480 seconds)
[18:00] * Tv_ (~tv@ Quit (Remote host closed the connection)
[18:01] * Tv_ (~tv@2607:f298:a:607:60e1:881b:ec67:50fe) has joined #ceph
[18:04] * dabeowulf (dabeowulf@free.blinkenshell.org) Quit (Remote host closed the connection)
[18:05] <NashTrash> If you do not specify "mon addr" does it just bind to
[18:06] * Leseb (~Leseb@ Quit (Quit: Leseb)
[18:07] * jamespage (~jamespage@tobermory.gromper.net) has joined #ceph
[18:08] * Cube (~Adium@cpe-76-95-223-199.socal.res.rr.com) Quit (Quit: Leaving.)
[18:12] * glowell (~Adium@c-98-210-226-131.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[18:18] * deepsa (~deepsa@ Quit (Ping timeout: 480 seconds)
[18:19] * deepsa (~deepsa@ has joined #ceph
[18:23] * tnt (~tnt@17.127-67-87.adsl-dyn.isp.belgacom.be) has joined #ceph
[18:23] <NashTrash> Sorry to pester, but if you do not specify "mon addr" does it just bind to
[18:25] <joshd> NashTrash: the monitors ip needs to be well known, since it's how clients connect. I don't remember the exact details, but I think it just picks an ip unless you specify one in your conf file
[18:26] * aliguori (~anthony@cpe-70-123-145-39.austin.res.rr.com) Quit (Quit: Ex-Chat)
[18:27] <joshd> in general you probably want to set mon addr in your ceph.conf file for each monitor
[18:29] * joshd (~jdurgin@2602:306:c5db:310:1e6f:65ff:feaa:beb7) Quit (Quit: Leaving.)
[18:30] * Cube (~Adium@ has joined #ceph
[18:32] * maelfius (~Adium@pool-71-160-33-115.lsanca.fios.verizon.net) has joined #ceph
[18:32] * maelfius (~Adium@pool-71-160-33-115.lsanca.fios.verizon.net) Quit ()
[18:33] * NashTrash (~Adium@mobile-166-147-119-148.mycingular.net) Quit (Ping timeout: 480 seconds)
[18:35] * deepsa (~deepsa@ Quit (Ping timeout: 480 seconds)
[18:36] * deepsa (~deepsa@ has joined #ceph
[18:37] * glowell (~Adium@c-98-210-226-131.hsd1.ca.comcast.net) has joined #ceph
[18:48] * bchrisman (~Adium@ has joined #ceph
[18:49] * aliguori (~anthony@ has joined #ceph
[18:52] * mgalkiewicz (~mgalkiewi@toya.hederanetworks.net) has joined #ceph
[18:57] * Ryan_Lane (~Adium@c-67-160-217-184.hsd1.ca.comcast.net) has joined #ceph
[19:05] * lurbs_ (user@uber.geek.nz) has joined #ceph
[19:07] * lurbs (user@uber.geek.nz) Quit (Ping timeout: 480 seconds)
[19:18] * chutzpah (~chutz@ has joined #ceph
[19:26] * maelfius (~Adium@ has joined #ceph
[19:29] * fc (~fc@ Quit (Quit: leaving)
[19:30] <mgalkiewicz> hi guys is it possible to do live resize of rbd volume?
[19:32] * The_Bishop (~bishop@2a01:198:2ee:0:140a:acf0:1ed9:84ce) has joined #ceph
[19:34] * joshd (~joshd@2607:f298:a:607:221:70ff:fe33:3fe3) has joined #ceph
[19:34] <mgalkiewicz> joshd: do you have a minute?
[19:35] <joshd> sure, what's up?
[19:35] <mgalkiewicz> is it possible to do live resize of rbd volume?
[19:35] <joshd> it is possible, but it might take some work to get various layers to acknowledge the resize
[19:36] <mgalkiewicz> I want to execute rbd resize vol_name on one of the mons and then resize filesystem on client side
[19:37] <joshd> you might have to force the kernel to refresh it's caches of the block device size
[19:37] <mgalkiewicz> how to do this?
[19:37] <joshd> there are some blockdev commands that do it iirc, I don't remember all the details off hand
[19:38] <mgalkiewicz> ok I will check wiki
[19:38] <mgalkiewicz> but this tree steps should be sufficient, shouldnt it?
[19:39] <joshd> I think so
[19:39] <joshd> oh, you might also need a newer kernel (assuming this is krbd)
[19:40] <mgalkiewicz> krbd?
[19:41] <joshd> kernel rbd
[19:41] <mgalkiewicz> I have but why do I need newer kernel?
[19:42] <mgalkiewicz> I am using 3.2.19 right now
[19:43] <joshd> yeah, the size wasn't actually updated live until recently...
[19:43] <iggy> tjpatter: you'd have to use a much newer version of qemu/kvm than rhel/centos provide
[19:43] <joshd> hmm, looks like that patch will be in 3.6
[19:44] <joshd> it was fixed back in december, but not integrated into the tree until recently
[19:44] <joshd> commit 93a24e084d67ba2fcb9a4c289135825b623ec864
[19:45] <mgalkiewicz> so it is not possible with older kernel than 3.6?
[19:45] <joshd> not unless you backport the fix
[19:45] <mgalkiewicz> shit
[19:46] <mgalkiewicz> so I need offline resize:/
[19:46] <mgalkiewicz> thx for help
[19:46] <joshd> sure
[19:50] <tjpatter> iggy: I just had a pretty extensive conversation in the CentOS channel. Looks like we would have to compile qemu-kvm from source.
[19:52] * mgalkiewicz (~mgalkiewi@toya.hederanetworks.net) Quit (Quit: Ex-Chat)
[19:53] <iggy> tjpatter: yeah, and some people have had issues with newer qemu-kvm on redhat's kernel
[19:55] <NaioN> sjust: the patch worked, thx
[19:55] <tjpatter> iggy: What about getting a .ko for the CentOS kernel to enable rbd and attaching via virsh disk-attach?
[19:55] <sjust> NaioN: good to hear!
[19:55] <NaioN> although I lost the rbd... but at least the cluster is in order
[19:56] <tjpatter> The CentOS kernel is new enough, but it doesn't come stock with the rbd module.
[19:56] <iggy> I wouldn't know anything about that... I'd be surprised if there was any kind of rbd backport for that old of a kernel
[19:57] <iggy> since there is no ceph backport and rbd uses a lot of the common ceph code
[19:57] <tjpatter> Hmm...
[19:58] <NaioN> sjust: i tried to remove the rbd but i have the feeling that didn't work well
[19:58] <NaioN> is there a way to look of objects still exist?
[20:03] * danieagle (~Daniel@ has joined #ceph
[20:14] <sjust> NaioN: From what I remember, the unfound object appeared to be the rbd index object
[20:14] <sjust> which is problematic
[20:15] <sjust> rados -p <poolname> ls
[20:15] <sjust> will give you the contents of the pool
[20:16] * BManojlovic (~steki@ has joined #ceph
[20:16] <NaioN> sjust: hmmm that explains
[20:16] <NaioN> it was complaining about the index
[20:16] <sjust> one sec, I may be able to get the prefix for you
[20:17] <NaioN> well it isn't a problem I lost the rbd (was backup) but now I've lost the space :)
[20:17] <sjust> once we have the prefix, we can find the actual object blocks and remove them manually
[20:17] <NaioN> hmmm something with edugrip vdr?
[20:18] <NaioN> or is it an id?
[20:18] <NaioN> hmmm an id i see :)
[20:19] <NaioN> well maybe i did something stupid, i tried to remove it and i didn't see the name anymore in rbd list
[20:20] <sjust> looks like the the head object was edugrip-vdr01.rbd
[20:20] <NaioN> and then i tried to create a new rbd with the same name
[20:20] <NaioN> but it failed
[20:20] <sjust> one sec
[20:20] <NaioN> yeah that's the one
[20:21] <sjust> crud, the block prefix names are not derivable from the rbd name
[20:21] <sjust> you'll need to compare the existing prefixes with the prefixes on existing rbd images
[20:22] <sjust> and remove the one that doesn't have a corresponding header
[20:22] <NaioN> well that's no problem
[20:22] <sjust> there is an rbd command to get the prefix for the blocks corresponding to an rbd header
[20:23] <NaioN> rbd info shows it
[20:23] <sjust> yeah
[20:23] <sjust> so rados ls will give you all of the existing objects in the pool, some will correspond to an existing rbd header, a few won't
[20:23] <NaioN> rbd image 'edugrip-vdr01b': size 1023 GB in 262127 objects order 22 (4096 KB objects) block_name_prefix: rb.0.4 parent: (pool -1)
[20:24] <sjust> that looks like a different header?
[20:24] <NaioN> this is the newly created rbd
[20:24] <sjust> ok
[20:24] <sjust> yeah
[20:24] <sjust> so don't remove anything starting with rb.0.4
[20:24] <NaioN> they needed to make new backups :)
[20:24] <sjust> ah
[20:24] <NaioN> yeah i get it
[20:24] <NaioN> there are more in that pool
[20:24] <NaioN> so i just look for all the active rbds
[20:25] <NaioN> luckily only a few rbds in that pool
[20:27] <NaioN> I have rbds with rbd.0.[0134] in that pool
[20:28] <NaioN> so I have to delete everything with rbd.0.2 (they also show up in the rados list)
[20:28] <NaioN> is there a command or do I have to make a one-liner to search all the osds and delete the files?
[20:29] <joshd> you can just grep the output of rados ls, piped to xargs rados rm
[20:30] <NaioN> aha ok
[20:30] <pmjdebruijn> put it through xargs echo first :D
[20:31] <NaioN> pmjdebruijn: why? :)
[20:31] <NaioN> don't trust me?
[20:32] <NaioN> sjust: thx! well i think i bashed the cluster a bit to hard... first time data loss :)
[20:33] <NaioN> pmjdebruijn: do you want it to do tomorrow at work together? :)
[20:37] <pmjdebruijn> NaioN: it's your call obviously :)
[20:39] <NaioN> well then tomorrow :)
[20:41] <NaioN> sjust: if i'm correct the only problem now is that till i remove the rb.0.2.* objects I can't make a new rbd, because it tries to use the rb.0.2 prefix, but it's already taken?
[20:41] * danieagle (~Daniel@ Quit (Quit: Inte+ :-) e Muito Obrigado Por Tudo!!! ^^)
[20:41] <NaioN> pmjdebruijn: yeah 48.1 released!
[20:42] * adjohn (~adjohn@m920536d0.tmodns.net) has joined #ceph
[20:45] <NaioN> pmjdebruijn: seems like the latest patch from sjust didn't make it into the 48.1 release after a quick scan in the changelog
[20:46] * adjohn (~adjohn@m920536d0.tmodns.net) Quit ()
[20:47] <maelfius> am I doing something wrong? I have a set of VMs running Ceph (1 running mon, 2 running an osd each). I shut everything down and when I restarted "ceph health" reports everything is ok, but rados ???pool rbd ls hangs as does all commands from rbd-cli tool. The VMs seem to be in good time sync. (this is on 0.48argonaut for 12.04 ubuntu)
[20:51] * adjohn (~adjohn@m920536d0.tmodns.net) has joined #ceph
[20:52] * loicd (~loic@brln-4dbab6b9.pool.mediaWays.net) Quit (Ping timeout: 480 seconds)
[20:53] <pmjdebruijn> NaioN: I'll apply it myself then
[20:53] <pmjdebruijn> no big deal
[20:53] <pmjdebruijn> sjust: I hope you will consider that patch we tested, to be queued up to 0.48.2 tho
[20:56] * adjohn (~adjohn@m920536d0.tmodns.net) Quit (Quit: adjohn)
[20:57] * loicd (~loic@brln-4dbab6b9.pool.mediaWays.net) has joined #ceph
[20:57] * tnt_ (~tnt@113.39-67-87.adsl-dyn.isp.belgacom.be) has joined #ceph
[20:59] * tnt (~tnt@17.127-67-87.adsl-dyn.isp.belgacom.be) Quit (Ping timeout: 480 seconds)
[21:11] <Tobarja> not following any lists right now, what's this sjust patch?
[21:17] <NaioN> Tobarja: we had a problem with a cluster recovering
[21:17] <NaioN> one pg had an unfound object and it was quering some osds for the unfound object
[21:18] <NaioN> but it tried to query 1 osd that didn't even have the pg on it and it returned a empty notify, the primary osd didn't deal with it and was still waiting
[21:19] <Tobarja> ok
[21:19] <NaioN> it wasn't on the list btw
[21:20] <lxo> sage, looks like our emails crossed. after my longer explanation, do you still need me to try to trigger the problem on master?
[21:21] <Tobarja> i set up ceph on one machine, but i've not had a chance to start up a second. can i play with data(a little bit) on it without it all of a sudden deciding it needs to replicate and freak out?
[21:21] <dmick> TIL "lxo" == "Oliver"
[21:22] <dmick> er, no, sorry, info actually helps :)
[21:22] <dmick> sorry Alexandre
[21:22] * NashTrash (~Adium@mobile-166-147-117-188.mycingular.net) has joined #ceph
[21:22] <NashTrash> Hello Cephers
[21:22] <Tobarja> in the past, i've blown up clusters in various ways :D
[21:26] <NaioN> Tobarja: you could set the replication level on 1
[21:28] <Tobarja> oh look, i found another box :) here goes nothin...
[21:41] <tjpatter> This goes back to a thread I had started here earlier, but on CentOS, I pulled down the latest kernel from elrepo and it includes the rbd module??? :)
[21:48] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[22:06] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[22:10] * nhorman (~nhorman@hmsreliant.think-freely.org) Quit (Quit: Leaving)
[22:42] <Tobarja> there is a page somewhere that says you can have as many X servers and 1 or 3+ Y servers, but I keep losing it. anyone know where that is?
[22:47] <NaioN> Tobarja: it's about the mons
[22:47] * Cube (~Adium@ Quit (Quit: Leaving.)
[22:47] <NaioN> they elect a leader for the cluster
[22:48] <Tobarja> mons can have as many as you want? or mds'es?
[22:48] <dmick> http://ceph.com/w/index.php?title=Designing_a_cluster&oldid=4153#Ceph_Monitor_.28Ceph-MON.29 has the best info I can find quickly
[22:48] <NaioN> Tobarja: both
[22:48] <mikeryan> mistyped "scrubber" as "scubber"
[22:48] <mikeryan> http://pbfcomics.com/20/
[22:48] <NaioN> but with the mons you have to have an uneven number
[22:48] <Tobarja> on that page, it says: Ceph Monitor... 2 is not recommended, you'll need to have both of them up all the time, worse than just having one.
[22:49] <Tobarja> but, that's the page i was looking for
[22:49] <NaioN> because in case of failure the cluster can only run if more than half of the mons are still connected to each other
[22:49] <NaioN> so they have the majority
[22:49] <NaioN> Tobarja: yeah indeed
[22:50] <NaioN> if you have two and one fails the other doesn't know if the other still lives and only the connection to the other failed or it really is dead
[22:51] <NaioN> with 3 if one fails or gets disconnected from the other two (but is still connected to the osds) the two elect the master and the third that got disconnected fences itself (removes itself from the cluster)
[22:51] <NaioN> because it never can be the master
[23:06] * adjohn (~adjohn@mb10536d0.tmodns.net) has joined #ceph
[23:07] <maelfius> (to anyone) when shutting down a cluster, is there anything specific that needs to be done. It seems as if when I shutdown (even an inactive cluster, no data) and start it back up, rbd commands rados commands etc that inspect the data, the commands hang indefinitely. Any advice?
[23:07] <dmick> shouldn't hang. How are you stopping/starting?
[23:08] <maelfius> service ceph stop <daemon>
[23:08] <maelfius> in the last test service ceph -a stop
[23:08] <maelfius> I cranked up debug_ms to 10, nothing really is standing out.
[23:08] <Tv_> maelfius: when you start it again, does the cluster become healthy?
[23:08] <maelfius> ceph health shows health_ok
[23:08] <maelfius> and ceph osd dump -o -
[23:09] <maelfius> shows everything looking good
[23:09] <maelfius> (unless i am misreading it)
[23:09] <maelfius> epoch 4
[23:09] <maelfius> fsid 10ff15e0-cee5-451a-bb90-7e7f37730c0c
[23:09] <maelfius> created 2012-08-14 21:04:44.539597
[23:09] <maelfius> modifed 2012-08-14 21:04:52.975980
[23:09] <maelfius> flags
[23:09] <maelfius> pool 0 'data' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num 192 pgp_num 192 last_change 1 owner 0 crash_replay_interval 45
[23:09] <maelfius> pool 1 'metadata' rep size 2 crush_ruleset 1 object_hash rjenkins pg_num 192 pgp_num 192 last_change 1 owner 0
[23:09] <maelfius> pool 2 'rbd' rep size 2 crush_ruleset 2 object_hash rjenkins pg_num 192 pgp_num 192 last_change 1 owner 0
[23:09] <maelfius> max_osd 3
[23:10] <maelfius> osd.1 up in weight 1 up_from 2 up_thru 3 down_at 0 last_clean_interval [0,0) exists,up 5bf65ded-8f34-4af5-8ab0-de9c0425a2eb
[23:10] <maelfius> osd.2 up in weight 1 up_from 3 up_thru 3 down_at 0 last_clean_interval [0,0) exists,up bcb53b55-bdec-4a8f-a629-8c2784a589
[23:10] <Tv_> maelfius: beware, the irc network will floodkick you soon
[23:10] <Tv_> maelfius: see also ceph -s
[23:11] <maelfius> tv_: yeah I wasn't planning on putting anything else in here. (I am aware of most flood kicks)
[23:11] * Cube (~Adium@cpe-76-95-223-199.socal.res.rr.com) has joined #ceph
[23:12] <maelfius> interesting, now it's showing 1 mon down
[23:12] <maelfius> but only 3 are defined and all are in.
[23:13] <dmick> network OK in all directions?
[23:14] <maelfius> dmick: yes. these are done on a single layer 2 segment no firewall rules (and I have verified connectivity to 2x check)
[23:15] <dmick> any hints in the down monitor's log about why it gave up?
[23:16] * aliguori (~anthony@ Quit (Quit: Ex-Chat)
[23:16] <maelfius> dmick: sec, looking to ee
[23:20] <maelfius> dmick: there seems to be something odd going on between two of the mons
[23:20] <maelfius> mon.0@0(leader) e1 handle_probe ignoring fsid 82ae1e59-9cda-4387-9f25-3fa13076adda != 10ff15e0-cee5-451a-bb90-7e7f37730c0c
[23:21] <maelfius> and mon.1@1(peon) has the same log line?
[23:21] * NashTrash (~Adium@mobile-166-147-117-188.mycingular.net) Quit (Quit: Leaving.)
[23:23] <maelfius> hrm. let me reinit that mon
[23:23] <maelfius> that is causing the issue.
[23:23] <dmick> yeah, that can't be good
[23:27] * adjohn (~adjohn@mb10536d0.tmodns.net) Quit (Quit: adjohn)
[23:32] * EmilienM (~EmilienM@ Quit (Quit: Leaving...)
[23:34] <maelfius> dmick: ok reinit of mon complete, same thing though, rbd ???pool rbd ls just hangs (there isn't any data in there, so it should return the "no images" message)
[23:34] <maelfius> oh and the cluster seems happy now, status HEALTH_OK
[23:36] <dmick> your command-line tools and your daemons match in version (you don't have a local build installed or something)?
[23:36] <maelfius> dmick: yes they match. I just did a clean install of 48.1
[23:36] <maelfius> (from the ubuntu packages)
[23:37] <dmick> and I assume the rados client also still hangs?
[23:38] <maelfius> dmick: yes. and to clarify lspools always worked, it's just when it goes to retrieve object data
[23:38] <maelfius> after my next meeting i can spin up some AWS instances and see if I have the same problem on a clean setup that way, if this seems like it shouldn't be happening
[23:38] * Tamil1 (~Adium@ has joined #ceph
[23:39] <dmick> no it definitely shouldn't be hanging, something's wrong
[23:40] <maelfius> dmick: the exact duplication I've had this happen with is: 1) spin up cluster, 2) verify it works, 3) shut down cluster, 4) startup cluster, 5) commands hang
[23:45] * lurbs_ (user@uber.geek.nz) Quit (Ping timeout: 480 seconds)
[23:48] <dmick> so I'd try client debug next, I guess
[23:49] <maelfius> dmick: i'll do so when i get back from my next meeting. thanks for your help. I'm going to tear down the boxes and restart to be sure I didn't do something wonky and repro + debug this before i come back.
[23:49] <dmick> add, say
[23:49] <dmick> debug ms = 1
[23:49] <dmick> debug client = 10
[23:49] <dmick> log file = <client log file location>
[23:49] <dmick> to [client] in ceph.conf
[23:49] <maelfius> dmick: will do
[23:50] <maelfius> again: ty! :)
[23:52] <dmick> yw
[23:55] * aliguori (~anthony@cpe-70-123-145-39.austin.res.rr.com) has joined #ceph
[23:56] * dspano (~dspano@rrcs-24-103-221-202.nys.biz.rr.com) Quit (Quit: Leaving)
[23:57] <sagewk> someone want to sanity-check wip-2918?

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.