#ceph IRC Log


IRC Log for 2012-11-14

Timestamps are in GMT/BST.

[0:02] * aliguori (~anthony@ Quit (Remote host closed the connection)
[0:06] * benner (~benner@ has joined #ceph
[0:06] * benner_ (~benner@ Quit (Read error: Connection reset by peer)
[0:08] * BManojlovic (~steki@ Quit (Quit: Ja odoh a vi sta 'ocete...)
[0:09] <sagewk> sjust: see wip-min-size
[0:09] <sagewk> ?
[0:11] <sjust> sagewk: so osd default min size = 0 means the default isn't set?
[0:11] <dmick> sagewk: tnx, will push to master
[0:11] <sjust> looks good
[0:11] <sagewk> yeah. not sure about those last two.. what do you think?
[0:12] <joshd> sagewk: it'd be nice to have a comment explaining what 0 means there
[0:13] * verwilst (~verwilst@dD5769628.access.telenet.be) Quit (Quit: Ex-Chat)
[0:13] <sagewk> joshd: good call
[0:14] <sjust> sagewk: min_size = 1 will effectively disable it
[0:14] <sjust> so using 0 for that seems reasonable
[0:14] * mtk (~mtk@ool-44c35983.dyn.optonline.net) Quit (Remote host closed the connection)
[0:15] <sagewk> k
[0:18] <sjust> benpol: stale means that the monitors haven't hard anything about that pg in a while
[0:18] <sjust> what does ceph -s currently say?
[0:19] * tnt (~tnt@18.68-67-87.adsl-dyn.isp.belgacom.be) Quit (Ping timeout: 480 seconds)
[0:21] <benpol> sjust: not seeing it now, but as I put my test cluster through chaos monkey style treatment (killiing OSDs etc) it shows up every now and then.
[0:21] * vata (~vata@ Quit (Remote host closed the connection)
[0:22] <sjust> benpol: if you kill all holders of a particular pg, that pg will be "stale" until one of them returns
[0:22] <sjust> and you won't see io to those objects
[0:22] <benpol> sjust: makes sense
[0:23] <lurbs> Which is why a CRUSH map that correctly maps your failure zones is necessary. :)
[0:24] <benpol> with three osd nodes and a default replication policy I wasn't expecting that turning off one of the three nodes would ever result in a :"stale-mate"
[0:24] <sjust> sagewk, glowell: master has a ceph-osd startup bug right now fyi, looking
[0:24] <benpol> lurbs: indeed!
[0:25] <sjust> benpol: turning off one actually shoudln't
[0:25] <sjust> *shouldn't
[0:26] <sjust> that would be a bug
[0:26] <sjust> can you describe how to reproduce the problem?
[0:26] <benpol> sjust: perhaps, I'll keep poking at it for a while
[0:27] <benpol> I suspect it'd be less likely to happen if I had more OSD nodes in the cluster.
[0:27] <sjust> yeah, but that doesn't mean it won't :)
[0:29] * benpol leans back in chair, taking pot shots at OSD nodes...
[0:32] <dmick> "jest like a-goin' rat huntin' down to the dump"
[0:33] <glowell> sjust: Should I be expecting a patch for 0.54 ?
[0:34] <sjust> oh, is 0.54 out already?
[0:34] <sjust> or wait
[0:34] <glowell> Just finished building. Hasn't been pushed yet.
[0:35] <sjust> ah, I got confused about the versions again, 0.54 isn't affected, it's something that went into master today
[0:36] * timmclaughlin (~timmclaug@ Quit (Ping timeout: 480 seconds)
[0:43] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[0:43] * loicd (~loic@magenta.dachary.org) has joined #ceph
[1:16] * miroslav (~miroslav@wlan-clients-8094.sc12.org) has joined #ceph
[1:16] * miroslav1 (~miroslav@wlan-clients-8094.sc12.org) Quit (Read error: Connection reset by peer)
[1:22] * Leseb (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) Quit (Quit: Leseb)
[1:23] * maxiz_ (~pfliu@ Quit (Quit: Ex-Chat)
[1:23] * maxiz (~pfliu@ Quit (Quit: Ex-Chat)
[1:31] * jlogan1 (~Thunderbi@2600:c00:3010:1:fc51:3370:98cf:1d11) Quit (Ping timeout: 480 seconds)
[1:34] * mtk (~mtk@ool-44c35983.dyn.optonline.net) has joined #ceph
[1:50] * yanzheng (~zhyan@ has joined #ceph
[1:50] * plut0 (~cory@pool-96-236-43-69.albyny.fios.verizon.net) has joined #ceph
[1:51] * timmclaughlin (~timmclaug@173-25-192-164.client.mchsi.com) has joined #ceph
[1:51] * timmclau_ (~timmclaug@173-25-192-164.client.mchsi.com) has joined #ceph
[1:51] * timmclaughlin (~timmclaug@173-25-192-164.client.mchsi.com) Quit (Read error: Connection reset by peer)
[1:55] * buck (~buck@bender.soe.ucsc.edu) has left #ceph
[1:59] <plut0> how does Gluster differ from Ceph?
[2:01] * miroslav (~miroslav@wlan-clients-8094.sc12.org) Quit (Quit: Leaving.)
[2:07] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[2:07] <lurbs> Gluster gives you a file system whereas most people, as far as I can tell, use Ceph for backing block devices. CephFS exists too, but isn't as mature as the RBD (RADOS Block Device) stuff.
[2:07] * loicd (~loic@magenta.dachary.org) has joined #ceph
[2:12] <plut0> ahh i see
[2:13] <sjust> lurbs, plut0: the cephFS part is where most of our active developement is going right now though!
[2:13] <lurbs> Pity, because that's the part I don't care about. :)
[2:14] <sjust> well, we are also actively developing rbd etc as well!
[2:14] <plut0> yeah i concur, i'd rather use another file system
[2:14] <sjust> plut0: why?
[2:14] <plut0> sjust: hard to find something comparable to zfs
[2:14] <sjust> oh, cephFS is a DFS
[2:15] <elder> If you don't need clustered access, a local file system is the better route.
[2:15] <sjust> we are talking about using zfs to back the osds, actually
[2:15] <sjust> right
[2:15] <plut0> yeah thats what i was thinking
[2:15] <plut0> so i'm on the right track then?
[2:15] <elder> But if you do need it, ceph fs has a lot going for it.
[2:15] <sjust> well, cephFS sits on the object store which sits on a local fs
[2:15] <sjust> the local fs might be zfs or something else
[2:16] <plut0> how else does it compare to gluster?
[2:17] <jtang> plut0: from my experiences at least with ceph you get a lot of HA stuff for free
[2:17] <jtang> where in glusterfs you need to setup linux ha to get it to be really available and redundent
[2:17] <plut0> jtang: gluster is commercialized?
[2:17] <jtang> redhat own glusterfs
[2:18] <plut0> jtang: you need to utilize heartbeat/pacemaker?
[2:18] <jtang> they sell support now (i asked a redhat rep today :))
[2:18] <plut0> is there commercial support for ceph?
[2:18] * `gregorg` (~Greg@ has joined #ceph
[2:18] * gregorg_taf (~Greg@ Quit (Read error: Connection reset by peer)
[2:18] <jtang> plut0: it depends on what your requirements are, but yea, if you are serious about running it in a production env, then yea
[2:19] <plut0> jtang: who supports it?
[2:19] <jtang> plut0: depending on how you layout your bricks in glusterfs you can get screwed over with not being able to expand
[2:19] * gohko_ (~gohko@natter.interq.or.jp) has joined #ceph
[2:19] <jtang> we ran into all sorts of issues with glusterfs when we were testing it
[2:20] <plut0> is there commercial support for ceph?
[2:20] <jtang> plut0: which gluster or cep?
[2:20] <jtang> ceph?
[2:20] <jtang> inktank support it?
[2:20] * jtang doesnt work for either companies
[2:21] <plut0> ok
[2:21] <joao> both inktank and 42on offer support services for ceph
[2:21] <sjust> Inktank does support for cephFS
[2:22] <jtang> though i forgot to ask the inktank people how it feels like to work for ceph
[2:22] <sjust> much fun
[2:22] <joao> makes me all tingly inside
[2:23] <elder> it's a privilege
[2:24] <jtang> i meant inktank
[2:24] <plut0> employees in the channel?
[2:24] <jtang> im stil on the fence about applying for a job
[2:24] * gohko (~gohko@natter.interq.or.jp) Quit (Ping timeout: 480 seconds)
[2:26] <jtang> heh 0.54 just got released
[2:26] <jtang> is it stable with cephfs?
[2:26] <sjust> somewhat, depends on the workloa
[2:26] <sjust> *workload
[2:27] * jtang reads the release notes
[2:27] * jtang wishes he could spend all day playing with ceph
[2:28] <jtang> but i have a day job
[2:28] <plut0> isn't it annoying how your day job gets in the way of your hobbies?
[2:28] <jtang> heh yea
[2:28] <joao> I haven't found a problem with that yet
[2:29] <joao> besides getting a couple more hobbies that is
[2:29] <jtang> well, we're looking at ceph in relation to a few projects in work
[2:29] <jtang> so i might get to play with it more soon
[2:29] <joao> got back to reading a whole lot in the last couple of months :p
[2:29] <jtang> well i get to play with it in 3rd person
[2:29] <plut0> jtang: what do you do for work?
[2:29] <jtang> gonna get someone else to set it up for me :)
[2:30] <jtang> plut0: i work in the area of digital preservation and archiving
[2:30] <plut0> jtang: and you'll get to play with ceph at work?
[2:31] <jtang> plut0: yea ;)
[2:31] <jtang> we need something to store our data with
[2:31] <jtang> and ceph is a candidate system for us right now
[2:32] <jtang> if it gets selected, i will be doing a bit more with it
[2:32] <plut0> jtang: you're lucky, most employers wouldn't allow that
[2:32] <jtang> well i am leading a small team
[2:32] <jtang> ;)
[2:32] <jtang> so i guess my boss trusts me enough to let me make decisions
[2:33] <jtang> which is a good thing
[2:33] <lurbs> Excellent, 0.54 fixed my bugs.
[2:33] <plut0> i work in health care where non-technical directors make poor decisions about IT
[2:33] <jtang> heh that sucks
[2:34] <jtang> luckily i dont deal with medical data these days
[2:34] <jtang> so i know how you feel
[2:34] <plut0> i don't even get to pick which SAN we use
[2:35] <plut0> it gets decided by the director who gets kick backs from some salesman
[2:35] <jtang> but you probably end up getting support contracts as well
[2:35] <jtang> so it cant be that bad?
[2:35] <lurbs> I have the opposite problem. The Powers That Be are dead set on open source, and open standard, and won't even start to consider a SAN.
[2:35] <lurbs> So it has to be something like Ceph.
[2:36] <jtang> heh
[2:36] <plut0> jtang: yes paid support but when its not architected correctly you feel much pain from a lot of performance problems and off hour phone calls
[2:36] <jtang> we're pretty pragmatic at our workplace, we're open to open and closed source solutions as long as it delivers our requirements
[2:36] <plut0> lurbs: so you rely on real engineering by intelligent people. not so bad but you still need paid support somewhere.
[2:38] <lurbs> For hardware, yes. For software, very difficult to convince them of.
[2:38] <lurbs> "Isn't that your job?", etc.
[2:38] <plut0> lurbs: yeah i've had those jobs too
[2:40] <plut0> ultimately you have to point the finger at a vendor when theres an issue that can't be solved
[2:45] <jtang> right time to go
[2:45] <jtang> last BoF session of the dayis over
[2:49] * sagelap (~sage@ Quit (Read error: Operation timed out)
[2:49] * maxiz (~pfliu@ has joined #ceph
[2:52] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[2:56] * mdawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) has joined #ceph
[2:57] * chutzpah (~chutz@ Quit (Quit: Leaving)
[3:13] * dmick (~dmick@2607:f298:a:607:1a03:73ff:fedd:c856) Quit (Quit: Leaving.)
[3:20] * timmclau_ (~timmclaug@173-25-192-164.client.mchsi.com) Quit (Remote host closed the connection)
[3:23] <glowell> Do we have a new dependency on libboost_thread ?
[3:32] * glowell (~glowell@c-98-234-186-68.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[3:44] * danieagle (~Daniel@ Quit (Quit: Inte+ :-) e Muito Obrigado Por Tudo!!! ^^)
[3:54] * Cube1 (~Cube@ Quit (Remote host closed the connection)
[4:01] * plut0 (~cory@pool-96-236-43-69.albyny.fios.verizon.net) has left #ceph
[4:03] * adjohn (~adjohn@ Quit (Quit: adjohn)
[4:08] * sjustlaptop (~sam@68-119-138-53.dhcp.ahvl.nc.charter.com) has joined #ceph
[4:12] <elder> I just rebuilt my ceph code and yes, I am now getting a build failure due to -lboost_thread
[4:12] <wilson> would using /dev/shm as the journal device be a fairly good way to determine the potential performance gains of using zeusram ssd for journal, I am thinking so? of course this is a dev environment its all garbage data
[4:17] * mdawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) Quit (Quit: ChatZilla 0.9.89 [Firefox 16.0.2/20121024073032])
[4:32] * mdrnstm (~mdrnstm@206-169-78-213.static.twtelecom.net) Quit (Quit: Leaving.)
[4:39] * yehudasa_ (~yehudasa@static-66-14-234-139.bdsl.verizon.net) has joined #ceph
[5:17] * yehudasa_ (~yehudasa@static-66-14-234-139.bdsl.verizon.net) Quit (Ping timeout: 480 seconds)
[5:22] * miroslav (~miroslav@ has joined #ceph
[5:22] * sjustlaptop (~sam@68-119-138-53.dhcp.ahvl.nc.charter.com) Quit (Ping timeout: 480 seconds)
[5:37] * kavonr (~rnovak@ns.indyramp.com) Quit (Quit: leaving)
[5:39] * rweeks (~rweeks@ has joined #ceph
[6:11] * LarsFronius (~LarsFroni@2a02:8108:380:12:49be:4e24:3b0:594a) Quit (Quit: LarsFronius)
[6:15] <jtang> hmmm
[6:15] <rweeks> hmmm
[6:15] <rweeks> ;)
[6:16] * jjgalvez1 (~jjgalvez@cpe-76-175-17-226.socal.res.rr.com) has joined #ceph
[6:18] <jtang> yea rweeks that was a good chat today
[6:18] <rweeks> I agree
[6:18] <jtang> i didnt get too talk about some stuff about how inktank is for careers
[6:18] <rweeks> I wrote your two big ideas up in email and sent them to Sage and others.
[6:19] <jtang> i was too focused on my current work
[6:19] * jjgalvez (~jjgalvez@cpe-76-175-17-226.socal.res.rr.com) Quit (Ping timeout: 480 seconds)
[6:19] <sage> apt-get install libboost-thread-dev
[6:19] <sage> elder, glowell: ^
[6:19] <jtang> i have a few ideas on extending the cepfs system
[6:20] <rweeks> well sage is right here, he might be the guy to talk to. :)
[6:20] <jtang> i really think there is scope for ceph in the archiving and preservation space
[6:20] <sage> not for long
[6:20] <sage> !
[6:20] <sage> jtang: yes
[6:20] <rweeks> I do too, jtang
[6:20] <jtang> heh cool, hi sage!
[6:20] <rweeks> sage, I sent you an email about jtang
[6:20] <rweeks> he has some good ideas
[6:20] <jtang> there is a gap right now in the archiving and preservation space
[6:21] <jtang> and i'd like to explore it more
[6:21] <rweeks> jtang: there is also a gap in digital media archives. we had some people from an LA media production company talking to us today about it
[6:22] <jtang> yea i have a friend who is in the sfx industry
[6:22] <jtang> and we're kinda thinking about the whole digital asset management issues
[6:22] <rweeks> yes
[6:22] <rweeks> these guys from LA have that problem too
[6:23] <rweeks> how do they retain and manage metadata about their productions
[6:24] <jtang> btw one of the other ideas that we've been thinking about is taking the crush library and possibly using in fedora-commons for data placement
[6:28] <jtang> and we have some ideas about creating microservices based on ceph
[6:28] <elder> sgae
[6:28] <elder> sage thanks I did figure that out.
[6:29] <jtang> we're pretty curious and interested in implementing irods styled policy engines and intelligent data
[6:31] <jtang> partly because we're also interested in possibly creating auditing systems as well
[6:31] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[6:31] <rweeks> that's some cool stuff, jtang
[6:31] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[6:33] <elder> :q
[6:34] <elder> Whoops.
[6:34] <elder> I guess that looks a bit smiley, but it was a vi command.
[6:34] <rweeks> [shift]zz
[6:34] <elder> Old Skool
[6:34] <rweeks> yup
[6:39] <jtang> rweeks: out of curiousity with the media companies that you talked to, are they current netapp or lustre users??
[6:39] <rweeks> the guys I talked to have "storage from everyone"
[6:39] <jtang> i know that a few companies in the uk are pretty big lustre users at the moment
[6:39] <rweeks> not lustre, though.
[6:39] <rweeks> but they were TV production, mostly
[6:39] <jtang> though i havent kept up with what they are doing
[6:40] <rweeks> this guy said they had netapp, isilon, nexsan, stornext, not sure what else
[6:41] <jtang> isilon are cool
[6:42] <jtang> their concept of parity at the block level is kinda cool
[6:42] * glowell (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) has joined #ceph
[6:42] <ramsay_za> I fully intend to have a ceremonial burning of my netapp once I have moved off it
[6:42] <rweeks> that will create quite a toxic fire
[6:42] <jtang> i havent seen too many solutions that do reed solomon coding
[6:42] <rweeks> isilon has cool concept yes
[6:42] <rweeks> in production… eh.
[6:43] <ramsay_za> yep it will but it will still be less toxic than the actual device
[6:43] <rweeks> without bashing all the storage vendors let's just say I am aware of most of their dirty laundry.
[6:43] <rweeks> disclosure: I worked for NetApp for 5 years.
[6:44] <ramsay_za> yeah, I could go on for weeks about all the crap my FAS has put us through
[6:45] <jtang> im never too sure about netapp
[6:45] <jtang> we never got any of their stuff in work
[6:45] * gregorg_taf (~Greg@ has joined #ceph
[6:45] * `gregorg` (~Greg@ Quit (Read error: Connection reset by peer)
[6:45] <jtang> they kinda suck at understanding the academic market
[6:46] <ramsay_za> in an office environment it's an awesome device but as backing for vms and cloud it fails horribly
[6:47] <jtang> ah okay, i wouldnt have thought netapp is good for that purpose anyway
[6:47] <rweeks> so you're moving to Ceph?
[6:48] <ramsay_za> going onto ceph and solidfire
[6:48] <ramsay_za> our vmware env will go onto solidfire our kvm onto ceph
[6:48] <rweeks> ah I see
[6:49] <elder> sage, are you online?
[6:50] <ramsay_za> jtang: in environments where you have idle time it works, but in a cloud env where it'll run hot for 6 months solid it all comes apart at the seams
[6:58] <jtang> ramsay_za: heh yea
[7:00] <elder> sage I'm going to bed, but I've sent my question by e-mail.
[7:16] <jtang> good night all, im off to bed!
[7:17] <ramsay_za> cheers, the day's just getting started this side of the pond
[7:18] * adjohn (~adjohn@108-225-130-229.lightspeed.sntcca.sbcglobal.net) has joined #ceph
[7:20] <jtang> heh ye i know im jetty lagged to bits
[7:24] * rweeks (~rweeks@ Quit (Quit: ["Textual IRC Client: www.textualapp.com"])
[7:26] * The_Bishop (~bishop@2001:470:50b6:0:2da4:d71a:ac6d:5cdb) has joined #ceph
[7:34] * glowell (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[7:38] <ramsay_za> rweeks: so where you working now that you have left netapp?
[7:54] <Robe> sage: Our new Dovecot Enterprise Release 2.2 has exciting new features, for example the possibility to support the features of the commercial module “Dovecot Object Storage Plugin”, which allows a high-performance connection to cloud storage systems, such as Amazon S3, Scality and CAStor from Caringo.
[8:24] * silversu_ (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[8:28] <iltisanni> Hey there.. I have a problem trying to mount with the following command: mount -t ceph /mnt/mycephfs -> I get the error FATAL: Module ceph not found. mount.ceph: modprobe failed, exit status 1 mount error: ceph filesystem not supported by the system
[8:29] <ramsay_za> what OS you running?
[8:29] <iltisanni> ubuntu 12.04
[8:29] <iltisanni> sry ubunutu 12.1
[8:30] <ramsay_za> on the client apt-get install ceph-common
[8:30] <ramsay_za> should do the trick
[8:30] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Ping timeout: 480 seconds)
[8:30] <iltisanni> i did that
[8:32] <ramsay_za> do an modprobe ceph
[8:33] <iltisanni> I did a kernel update to 3.6.3 because I tried the same command before with kernel 3.5317 and suddenly everything freezed and I had to hard shutdwon the Client VM...so I did a kernel upgrade (somebody here told me that could work)
[8:34] <iltisanni> modprobe ceph -> module ceph not found
[8:34] <ramsay_za> do a locate ceph.ko and check that you have a ceph module in the folder for your current kernel i.e. /lib/modules/3.2.0-2-amd64/kernel/fs/ceph/ceph.ko
[8:35] <iltisanni> OK.. I havent
[8:35] <ramsay_za> might need an updatedb to refresh the cache
[8:36] <iltisanni> it was in the old kernels directory -> /lib/modules/3.5.0-17-generic/kernel/fs/ceph/ceph.ko
[8:37] <iltisanni> updatedb dind resolve the problem..
[8:37] <iltisanni> Can I just copy the ceph.ko ?
[8:37] <ramsay_za> updatedb just updates the locate cache
[8:38] <dweazle> depmod -a
[8:38] <ramsay_za> you can try just copying it but to be safe I'd build and install the kernel module again
[8:39] <iltisanni> what is depmod for ? (sorry; I'm new to linux)
[8:39] <dweazle> man depmod ;)
[8:39] <iltisanni> y right :-)
[8:41] * miroslav (~miroslav@ Quit (Quit: Leaving.)
[8:45] * loicd (~loic@magenta.dachary.org) has joined #ceph
[8:48] * adjohn (~adjohn@108-225-130-229.lightspeed.sntcca.sbcglobal.net) Quit (Quit: adjohn)
[8:49] * tnt (~tnt@18.68-67-87.adsl-dyn.isp.belgacom.be) has joined #ceph
[8:52] * silversu_ (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[8:55] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[8:55] * loicd (~loic@magenta.dachary.org) has joined #ceph
[9:00] * verwilst (~verwilst@d5152D6B9.static.telenet.be) has joined #ceph
[9:02] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[9:07] * BManojlovic (~steki@ has joined #ceph
[9:16] <gucki> good morning :)
[9:16] <gucki> someone knows how to get the python bindings to work? which packages etc. do I have to install?
[9:29] * tnt (~tnt@18.68-67-87.adsl-dyn.isp.belgacom.be) Quit (Ping timeout: 480 seconds)
[9:36] <gucki> ok, found it :
[9:37] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has joined #ceph
[9:37] <ramsay_za> cool, yeah I have zero experience in that area
[9:38] <ramsay_za> share with the class, so the logs have the answer for anyone that get's to it via a google search, please
[9:38] * jbd_ (~jbd_@34322hpv162162.ikoula.com) has left #ceph
[9:40] * jbd_ (~jbd_@34322hpv162162.ikoula.com) has joined #ceph
[9:42] * tnt (~tnt@212-166-48-236.win.be) has joined #ceph
[9:55] * ScOut3R (~ScOut3R@catv-80-98-44-93.catv.broadband.hu) has joined #ceph
[9:57] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[10:06] * SIN (~SIN@ has joined #ceph
[10:13] * mistur (~yoann@kewl.mistur.org) Quit (Ping timeout: 480 seconds)
[10:20] * kees_ (~kees@devvers.tweaknet.net) has joined #ceph
[10:21] * mistur (~yoann@kewl.mistur.org) has joined #ceph
[10:21] <SIN> Hello! Can any one help me? I can`t understand 'ceph mon add' syntax. In the official documentation at the end of this line '\n";' What does it means?
[10:23] <SIN> Here is a link http://ceph.com/docs/master/cluster-ops/add-or-rm-mons/ Step 7 string
[10:23] <kees_> \n is a newline, i guess it it there so you get newlines in your config
[10:26] <kees_> in this case it looks more like a code fuckup tho, so i'd try that command without the \n";
[10:27] * loicd (~loic@ has joined #ceph
[10:27] <tnt> Be careful ... switching from 1 mon to 2 mon using those command is a sure way to shutdown the cluster ....
[10:28] <tnt> switching from 2 to 3, 3 to 4, 4 to 5 ... etc is OK, but 1 to 2 is a bit special case.
[10:29] <SIN> Any advice?
[10:30] <SIN> I got only 1 mon now
[10:31] <tnt> don't issue the mon add. Just start the second MON demon and let it start and joing the cluster first.
[10:33] * The_Bishop (~bishop@2001:470:50b6:0:2da4:d71a:ac6d:5cdb) Quit (Remote host closed the connection)
[10:38] * LarsFronius (~LarsFroni@testing78.jimdo-server.com) has joined #ceph
[10:40] <SIN> Ok, second mon started. Do I 'mon add' command? Or just leave it?
[10:42] <tnt> pastebin a ceph mon status
[10:44] * maxiz (~pfliu@ Quit (Quit: Ex-Chat)
[10:45] <SIN> http://pastebin.com/0mjR1UkF
[10:46] <tnt> no need for the mon add, it's been properly added AFAICT.
[10:46] <tnt> and btw, you might want to put a firewall :p
[10:47] <SIN> thanks
[10:50] * tryggvil (~tryggvil@16-80-126-149.ftth.simafelagid.is) Quit (Quit: tryggvil)
[10:56] <ctrl> hi all!
[10:56] <ctrl> i have problem with ceph (
[10:57] <ctrl> when i switch off second node, cluster hangs
[10:57] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) has joined #ceph
[10:57] * MikeMcClurg (~mike@firewall.ctxuk.citrix.com) has joined #ceph
[10:58] <ctrl> anybody?
[10:58] <joao> here now
[10:58] <andreask> you mean you loose quorum?
[10:58] <joao> morning #ceph
[10:59] <ctrl> andreask: i think so
[10:59] * Leseb (~Leseb@ has joined #ceph
[11:00] <andreask> ctrl: well, then it is working as expected
[11:00] <joao> SIN, good catch on that '\n'
[11:00] <joao> will fix that now
[11:01] <ctrl> andreask: no, hangs and hangs
[11:02] <ctrl> andreask: back to life, when second node is online
[11:02] <tnt> what do you mean by 'node' ?
[11:02] <tnt> there is mon, osd, mds ...
[11:02] <andreask> ctrl: you need at least 3 mons ...
[11:03] <ctrl> i have 2 nodes, mon mds osd per node
[11:03] <tnt> if you have 2 mon and you shutdown one, then yes, it will hang, that's expected.
[11:03] <tnt> you always need (floor(N/2) + 1) mon online.
[11:03] <tnt> so with 2 mon ... you always need both of them up.
[11:04] <tnt> see the 'mon' section of http://ceph.com/wiki/Designing_a_cluster
[11:05] <tnt> you basically need an odd number of MONs
[11:07] <ctrl> in "Cluster Design Recommendations" : If you have just a few nodes, you can put Ceph-MON, Ceph-MDS, and Ceph-OSD on the same node.
[11:08] <joao> tnt, that's a good rule of thumbs and good practice, but you don't *always* need an odd number of mons
[11:08] <joao> I mean, the cluster will work with an even number of mons
[11:08] <tnt> that's why I said 'basically' ...
[11:08] <joao> oh, sorry
[11:09] <joao> still waking up and clearly reading only half of what's on the screen
[11:09] <tnt> ctrl: well you need to run the entire page, not just one sentence.
[11:10] <tnt> if you have only 2 machines, then your only choice is 1 mon and make sure it doens't go down.
[11:11] <joao> yay, managed to update the docs without screwing other formatting
[11:11] <ctrl> ok. what`s about failover?
[11:12] <tnt> can't really do it with 2 nodes
[11:12] <tnt> you need 3
[11:13] <joao> ctrl, you're safer with only one mon than with two
[11:13] <tnt> (well you could setup something with drbd and pacemaker but that gets messy)
[11:13] <joao> at least it will avoid you headaches when one of the nodes fail
[11:14] <joao> you'll know the cluster just doesn't work because, well, the monitor failed
[11:15] <joao> on the other hand, if the mon node didn't fail, you'll have a working cluster (as long as the osds and mds are able to cope with the second monitor being down)
[11:15] <ctrl> ok. i will try with 1 mon. thx for answers!
[11:16] <joao> with a two node/two mon setup, regardless of the node that fails, you'll end up without a working cluster
[11:16] <joao> so there's that
[11:16] <joao> availability driven by luck
[11:21] <kees_> wido, am i correct to assume that phprados is kinda dead? won't compile for me on ubuntu 12.04 or 12.10 and i don't see much activity at git
[11:23] <fmarchand> Hi everybody !
[11:23] <fmarchand> Hi joao !
[11:24] <joao> morning fmarchand :)
[11:25] <fmarchand> I have a memory issue I think ... my mds take 1.3Gb .... does it look "normal" ?
[11:25] <fmarchand> and my two osd take 900Mb each ...
[11:25] <fmarchand> And I don't know if it's normal or not ...
[11:25] <joao> couldn't say if it is "normal" as I'm not involved in the mds, but given the guys are still working on it I'm not that surprised
[11:26] <joao> fwiw, there have been some extra effort in the last couple of days to fix memleaks throughout ceph
[11:27] <joao> fmarchand, my test osd was taking about that much yesterday, almost brought my desktop to a halt with all the thrashing, so I'm not surprised
[11:27] <joao> the osd can spike on mem usage some times, afaik
[11:28] <joao> (although in my case I'm also considering bad ram)
[11:28] * yanzheng (~zhyan@ Quit (Remote host closed the connection)
[11:36] * jjgalvez1 (~jjgalvez@cpe-76-175-17-226.socal.res.rr.com) Quit (Quit: Leaving.)
[11:37] <fmarchand> you're a developer ?
[11:38] <fmarchand> now my mds takes 1.4Gb ...
[11:39] <fmarchand> I suppose someone is adding files to the cluster
[11:39] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[11:39] <joao> fmarchand, yeah
[11:40] <fmarchand> wow ! So you know a bit about CRUSH algorithm :)
[11:40] <joao> a bit, yes
[11:41] <joao> might need to grab some more coffee first if you want to talk about that
[11:41] <fmarchand> yes I tried to read the publication describing it .... coffee may not be enough strong :) It was not for me !
[11:42] <fmarchand> But it's a cool thing ... I like the concept of decentralized stuff
[11:44] <fmarchand> Oh by the way ... I don't know if you saw yesterday my question ... but I have a weird thing about removed folders that come back in the nigh :) And no I can't remove them ...
[11:45] <fmarchand> now
[11:45] <fmarchand> It tells me it's not empty even with a "rm -rf"
[11:46] <joao> yeah, I saw that
[11:46] <joao> was it on a mounted cephfs volume?
[11:46] <fmarchand> I saw there was a issue opened about something like that but I'm not sure
[11:46] <fmarchand> yes it was
[11:47] <fmarchand> I mounted it back this morning and my zombie folders were back !
[11:48] <joao> you should try the list or catch slang around later in the afternoon
[11:48] <joao> I honestly have no idea what to do about that
[11:48] <fmarchand> it looks like a mds thing
[11:49] <joao> probably
[11:49] <fmarchand> slang .. oki thanks
[11:49] <joao> if he's not the right person to talk about this, I'm sure he'll point you out to the right one :)
[11:50] <joao> that said, going to brew some more coffee; bbiab
[11:51] <fmarchand> have a nice coffee !
[12:06] * MikeMcClurg (~mike@firewall.ctxuk.citrix.com) Quit (Ping timeout: 480 seconds)
[12:14] * SIN (~SIN@ Quit (Quit: Leaving)
[12:20] * gregorg_taf (~Greg@ Quit (Read error: Connection reset by peer)
[12:27] * gregorg (~Greg@ Quit (Ping timeout: 480 seconds)
[12:47] * MikeMcClurg (~mike@firewall.ctxuk.citrix.com) has joined #ceph
[13:48] * maxiz (~pfliu@ has joined #ceph
[13:57] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has left #ceph
[14:23] <fmarchand> Slang ? are you there ?
[14:23] * slang waves
[14:23] <fmarchand> Hi slang !
[14:23] <slang> fmarchand: hi!
[14:24] * slang reads scrollback
[14:24] <fmarchand> I'm in france so I didn't if you were there
[14:24] <fmarchand> You're in US, right ?
[14:25] <slang> fmarchand: yep
[14:25] <slang> fmarchand: so you have zombie folders?
[14:26] <fmarchand> I was about to tell you that :)
[14:26] <fmarchand> yes I have ...
[14:26] <slang> fmarchand: are you using ceph-fuse or the cephfs kernel module?
[14:27] <fmarchand> I use cephfs. I din't try ceph-fuse ... and don't know how to use it
[14:27] <fmarchand> I put a line in my fstab to mount ceph fs and that's it
[14:28] <fmarchand> pointing on my mon ip and port
[14:29] <fmarchand> I'm a newbee to ceph so ...
[14:29] <fmarchand> I'm afraid I did something wrong
[14:30] <slang> fmarchand: oh its probably a bug
[14:30] <slang> fmarchand: anyway, can you list the zombie folders?
[14:42] * rongze (~zhu@ has joined #ceph
[14:44] <elder> Anybody seen this before in a teuthology run?
[14:44] <elder> INFO:teuthology.task.ceph.osd.3.err:daemon-helper: command crashed with signal 9
[14:44] <elder> Does that mean someone up and killed my job?
[14:44] <slang> probably
[14:45] <slang> elder: when was that?
[14:45] <elder> That helps a lot.
[14:45] <elder> Last night.
[14:45] <elder> Well, sometime after 10:35:04 PSD
[14:45] <elder> PST
[14:47] <elder> Probably about 11pm PST.
[14:47] <slang> elder: some jobs were killed yesterday afternoon, but it was way before that
[14:48] * timmclaughlin (~timmclaug@ has joined #ceph
[14:48] <elder> It doesn't really matter much to me, I was running a very long test and it was doing fine so I already got my confidence out the exercise. I just wanted to understand the problem.
[14:50] * senner (~Wildcard@68-113-232-90.dhcp.stpt.wi.charter.com) has joined #ceph
[14:57] * aliguori (~anthony@cpe-70-123-145-75.austin.res.rr.com) has joined #ceph
[15:00] <iltisanni> Hey. I did a kernel update on all of my cluster VMs. After that I rebooted them. Suddenly my crushmap was totally empty !! All pgs in stuck stale mode.... Is that a known issue? Or did someone else have similar problems? I know that restarting all cluster machines at the same time is not the best idea but killing the whole cluster for that is not okay :-)
[15:02] * verwilst (~verwilst@d5152D6B9.static.telenet.be) Quit (Read error: Operation timed out)
[15:03] <tnt> huh ... the kernel on the vm shouldn't matter much to the crushmap ...
[15:03] <tnt> or maybe you're using btrfs and it screwed up the filesystem but then it's more btrfs fault's than ceph's.
[15:05] * nhorman (~nhorman@hmsreliant.think-freely.org) has joined #ceph
[15:11] <iltisanni> well ok . I'll try to reproduce this "feature" ;-)
[15:13] * styx-tdo (~styx@chello084113243057.3.14.vie.surfer.at) Quit (Ping timeout: 480 seconds)
[15:19] * verwilst (~verwilst@d5152D6B9.static.telenet.be) has joined #ceph
[15:21] <fmarchand> slang : sorry a meeting ... yes I can't list them but I can't remove them anymore
[15:22] <fmarchand> sorry I CAN list them
[15:22] <fmarchand> I can rename them as well
[15:29] * LarsFronius_ (~LarsFroni@testing78.jimdo-server.com) has joined #ceph
[15:29] * LarsFronius (~LarsFroni@testing78.jimdo-server.com) Quit (Read error: Connection reset by peer)
[15:29] * LarsFronius_ is now known as LarsFronius
[15:31] * LarsFronius (~LarsFroni@testing78.jimdo-server.com) Quit (Remote host closed the connection)
[15:31] * LarsFronius (~LarsFroni@testing78.jimdo-server.com) has joined #ceph
[15:33] <slang> fmarchand: have you created any snapshots?
[15:33] * joao (~JL@ Quit (Remote host closed the connection)
[15:34] <elder> slang, ahaaaa!! One of my nodes ran out of memory!
[15:34] <slang> elder: :-)
[15:34] <elder> The "somebody" that killed my process was the kernel.
[15:34] <slang> elder: user oom-killer did it?
[15:34] <elder> [371476.274991] Out of memory: Kill process 11092 (ceph-osd) score 456 or sacrid
[15:35] <elder> [371476.338586] Killed process 11092 (ceph-osd) total-vm:8259960kB, anon-rss:35B
[15:35] <elder> Looks like ceph-osd has a memory leak. (I think that's what that means. It had 8 TB of virtual memory)
[15:35] <fmarchand> slang : I knew about snapshots (I mean I know the feature exists) but I don't know how to do that (not yet) so no snapshots
[15:35] <slang> fmarchand: ok just checking
[15:36] <elder> sjust, sage ^^
[15:37] * dspano (~dspano@rrcs-24-103-221-202.nys.biz.rr.com) has joined #ceph
[15:38] <slang> fmarchand: can you do: ceph mds tell 0 injectargs --debug_mds 20
[15:39] <slang> fmarchand: then try to remove the zombie directory
[15:39] <fmarchand> oki
[15:39] <fmarchand> done
[15:40] <fmarchand> I guess you want my logs ...
[15:40] <slang> fmarchand: then post the output of your /var/log/ceph/mds.0.log
[15:41] <slang> fmarchand: to pastebin or somewhere accessible?
[15:41] <slang> fmarchand: the file might be relatively large, so you won't want to paste it into irc :-)
[15:42] <slang> fmarchand: or you can just email it to me (sam.lang@inktank.com)
[15:42] <fmarchand> yes :) but before in the mds.a.log I have : 1 mds.0.12 Ignoring empty injectargs!
[15:42] <slang> ooh
[15:43] <slang> fmarchand: put single quotes around the --debug_mds 20 i.e. '--debug_mds 20'
[15:44] <fmarchand> mmm oki
[15:44] <slang> fmarchand: oh I got the name wrong
[15:45] <fmarchand> that's better
[15:45] <slang> fmarchand: its: ceph mds tell a injectargs '--debug_mds 20'
[15:45] <slang> fmarchand: you should see a bunch of output to the log if the command works
[15:46] <fmarchand> oki, you want me to truncate the log file before ? it's pretty big right now
[15:46] <fmarchand> but the first one you told me showed me a bunch of ouput in the log
[15:47] <slang> fmarchand: you don't have to truncate, no
[15:47] <slang> fmarchand: what's the name of the dir you're trying to remove?
[15:49] <fmarchand> 2012-11-09TOREMOVE
[15:49] <slang> fmarchand: k
[15:50] <fmarchand> 162.9 MB ..... mmmm
[15:51] <fmarchand> slang : a bit too big I guess
[15:52] <slang> you can gzip it
[15:53] * fmarchand (~fmarchand@ Quit (Remote host closed the connection)
[15:54] * nosebleedkt (~kostas@kotama.dataways.gr) has joined #ceph
[15:54] * joao (~JL@ has joined #ceph
[15:54] * ChanServ sets mode +o joao
[15:57] * fmarchand (~fmarchand@ has joined #ceph
[15:58] <fmarchand> re
[15:58] <fmarchand> sorry my machine crached
[15:59] <slang> fmarchand: woops
[15:59] <fmarchand> it happens when you want to keep using compiz ....
[16:00] * slang nods
[16:00] <jtang> dont use compiz
[16:00] <jtang> :)
[16:05] * noob2 (a5a00214@ircip3.mibbit.com) has joined #ceph
[16:05] <fmarchand> I know ...
[16:06] <fmarchand> slang : is it normal that the file is getting really big ?
[16:06] <fmarchand> I mean the log file
[16:07] <slang> fmarchand: yeah the default logging is relatively verbose right now, you can setup logrotate to keep it from chewing up too much space
[16:08] <fmarchand> it's now ... 800 Mb ...
[16:08] * nosebleedkt (~kostas@kotama.dataways.gr) Quit (Quit: Leaving)
[16:08] <fmarchand> doing nothing
[16:09] * fmarchand (~fmarchand@ Quit (Remote host closed the connection)
[16:09] <slang> fmarchand: debug_mds 20 is *very* verbose
[16:10] * styx-tdo (~styx@chello084114016228.14.vie.surfer.at) has joined #ceph
[16:11] * styx-tdo is now known as Guest5616
[16:13] * tryggvil_ (~tryggvil@rtr1.tolvusky.sip.is) has joined #ceph
[16:13] * Guest5616 (~styx@chello084114016228.14.vie.surfer.at) Quit (Read error: Connection reset by peer)
[16:14] * styx_ (~styx@chello084114016228.14.vie.surfer.at) has joined #ceph
[16:16] * fmarchand (~fmarchand@ has joined #ceph
[16:17] <fmarchand> mmm ... I love to reboot every 5 min
[16:17] <fmarchand> re slang : the file takes 7Mb it's ok ?
[16:18] * styx_ (~styx@chello084114016228.14.vie.surfer.at) Quit (Read error: Connection reset by peer)
[16:18] <slang> fmarchand: debug_mds 20 is very verbose, once you have the logs from trying to remove the dir, you can disable it with the same injectargs command but --debug_mds 0
[16:18] <slang> fmarchand: yep that's fine
[16:18] * styx_ (~styx@chello084114016228.14.vie.surfer.at) has joined #ceph
[16:18] <fmarchand> oki
[16:20] <fmarchand> sorry but where should I send it ? sam...@inktank.org ...
[16:20] <fmarchand> ?
[16:21] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) Quit (Ping timeout: 480 seconds)
[16:22] * tryggvil_ (~tryggvil@rtr1.tolvusky.sip.is) Quit (Ping timeout: 480 seconds)
[16:22] <slang> fmarchand: sam.lang@inktank.com
[16:25] <joao> well, I better start backing up everything
[16:25] <joao> got to get a new hdd still this week
[16:26] <joao> third time in two days that had to hard reboot the desktop due to dma read errors
[16:26] <noob2> so i've got a networking question for you guys :)
[16:26] <noob2> if i setup osd servers with 4x 1Gb connections will ceph take advantage of that with multiple replication connections? If it only opens up one connection I think it can't span multiple network slaves
[16:27] <fmarchand> slang : sent ! I checked the log file .... you're right this is really verbose !
[16:27] <tnt> The OSD will have different IPs so it will spread across links.
[16:27] <tnt> noob2: ^^
[16:28] <match> noob2: See also http://www.sebastien-han.fr/blog/2012/07/29/tip-ceph-public-slash-private-network-configuration/ for ideas about splitting 'inter-osd' traffic, and client traffic
[16:30] * kees_ (~kees@devvers.tweaknet.net) Quit (Remote host closed the connection)
[16:32] <noob2> ok thanks guys i'll check that out
[16:32] <noob2> i didn't know if i needed to upgrade to 10Gb yet. We don't have jumbo frames here yet so I only get about 6Gb on my 10Gb lines. 4x1 comes close :)
[16:32] <fmarchand> joao : yes ... I had this kind of IO error and it crashed after one week ...
[16:33] <joao> yeah, that's what I'm afraid of
[16:33] <fmarchand> joao : you should use a 5*osd 5*mds 5*mon ceph cluster for your backup ;)
[16:33] * styx__ (~styx@chello084113243057.3.14.vie.surfer.at) has joined #ceph
[16:35] <match> noob2: I'd look into bonding those lines and using the public/private trick... though it all comes down to your disk speed - no advantage to shunting more network data if your disk io won't keep up :)
[16:35] <joao> fmarchand, I should indeed
[16:35] * styx_ (~styx@chello084114016228.14.vie.surfer.at) Quit (Read error: Operation timed out)
[16:36] <joao> I'm thinking about giving myself a home ceph cluster for Christmas :p
[16:36] <noob2> match: yeah exactly.
[16:37] <fmarchand> joao : oh yeah ! I'm gonna ask my wife for the same ! It's too bad that you can't boot from a ceph cluster :)
[16:37] <fmarchand> slang : did you receive my file ?
[16:37] <joao> well, you could with a thin client and rbd
[16:37] <fmarchand> slang : by the way I don't use auth
[16:39] * gaveen (~gaveen@ has joined #ceph
[16:43] * scalability-junk (~stp@188-193-211-236-dynip.superkabel.de) has joined #ceph
[16:54] * styx__ (~styx@chello084113243057.3.14.vie.surfer.at) Quit (Ping timeout: 480 seconds)
[17:03] * tryggvil (~tryggvil@16-80-126-149.ftth.simafelagid.is) has joined #ceph
[17:04] <fmarchand> tiny question : I if I have 2 1T osd's ... can I add a 200Go osd ? should I use a weight to avoid having the new osd full after the rebalancing ?
[17:04] <fmarchand> 200Gb
[17:06] <tnt> I'm curious too ... currently mine are more or less balanced, but when adding new servers I might not have the same hdd size per osd ...
[17:06] <ScOut3R> match: thanks for the link, just what i was looking for :)
[17:07] <match> ScOut3R: no problem - I found out about that feature when someone else posted that link here a while ago too :)
[17:08] * styx__ (~styx@chello084113243057.3.14.vie.surfer.at) has joined #ceph
[17:10] <fmarchand> I really don't know what to do ... I have 2 osd's on the same machine and they take 2.4 Gb in RAM ....
[17:11] <fmarchand> and it's increasing ...
[17:11] <tnt> running the 0.48.2 ?
[17:11] <fmarchand> yep
[17:11] <tnt> strange, here they're pretty stable at ~ 300 Mo.
[17:12] <fmarchand> even with millions of files ?
[17:12] <tnt> I don't use cephfs, only rbd and radosgw
[17:12] <fmarchand> I have a lot of small files ... I guess that's why ...
[17:13] <slang> fmarchand: got your file, thanks!
[17:13] <tnt> Here I have ~ 500k rados objects ...
[17:13] <slang> fmarchand: still looking at it...
[17:14] <fmarchand> thx slang ! I hope you'll find something
[17:14] <fmarchand> sorry but .... rbd is a fs ?
[17:15] <tnt> no
[17:16] <fmarchand> rados block driver ?
[17:16] <tnt> yes.
[17:16] <tnt> It's not a fs. You can put a fs on it, but it's not a fs.
[17:17] <fmarchand> I use ext4 .... maybe that's why
[17:17] <fmarchand> how can you know the number of inode across all your osd's ? is there a command
[17:17] <fmarchand> ?
[17:18] <tnt> huh ? I never said anything about inodes.
[17:19] <tnt> using 'rados df' you can see the underlying rados objects
[17:20] <fmarchand> no no I mean I guess it's taking so much ram because I have too many small files ... so I would like to know how many files I have on my osd's ...
[17:21] <fmarchand> I thought the mds could give that kind of information (for directories at least)
[17:25] * ivoks (~ivoks@jupiter.init.hr) Quit (Remote host closed the connection)
[17:25] <fmarchand> I think choosing ext4 as fs for osd's was a bad idea ... ?
[17:25] * ivoks (~ivoks@jupiter.init.hr) has joined #ceph
[17:28] * ivoks (~ivoks@jupiter.init.hr) Quit ()
[17:28] * ivoks (~ivoks@jupiter.init.hr) has joined #ceph
[17:28] * ivoks (~ivoks@jupiter.init.hr) Quit ()
[17:28] * ivoks (~ivoks@jupiter.init.hr) has joined #ceph
[17:30] * verwilst (~verwilst@d5152D6B9.static.telenet.be) Quit (Quit: Ex-Chat)
[17:38] * BManojlovic (~steki@ Quit (Quit: Ja odoh a vi sta 'ocete...)
[17:40] * jlogan1 (~Thunderbi@2600:c00:3010:1:fc51:3370:98cf:1d11) has joined #ceph
[17:41] * sagelap (~sage@232.sub-70-197-145.myvzw.com) has joined #ceph
[17:41] * sagelap (~sage@232.sub-70-197-145.myvzw.com) has left #ceph
[17:41] * sagelap (~sage@232.sub-70-197-145.myvzw.com) has joined #ceph
[17:42] * ScOut3R (~ScOut3R@catv-80-98-44-93.catv.broadband.hu) Quit (Remote host closed the connection)
[17:51] <noob2> tnt: is it correct in assuming that 1 osd process opens one network connection to the other osd's?
[17:51] * MikeMcClurg (~mike@firewall.ctxuk.citrix.com) Quit (Remote host closed the connection)
[17:52] <tnt> I think so yes.
[17:52] <noob2> ok
[17:52] <tnt> At least to other osd it shares PG's with
[17:52] <noob2> that seems to match up with what netstat is saying
[17:52] <noob2> right
[17:55] <noob2> 6802 and 6805 are the ports it seems to talk over
[18:02] * tnt (~tnt@212-166-48-236.win.be) Quit (Read error: Operation timed out)
[18:05] * sagelap (~sage@232.sub-70-197-145.myvzw.com) Quit (Ping timeout: 480 seconds)
[18:07] <noob2> so i think to resolve my potential network bandwidth problems i should just add more osd's per machine to open more connections
[18:15] * sagelap (~sage@2600:1013:b017:7c26:d13b:da08:ecfe:a0e4) has joined #ceph
[18:16] * bchrisman (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[18:17] * tnt (~tnt@18.68-67-87.adsl-dyn.isp.belgacom.be) has joined #ceph
[18:22] * sagelap1 (~sage@2607:f298:a:607:d13b:da08:ecfe:a0e4) has joined #ceph
[18:23] * sagelap (~sage@2600:1013:b017:7c26:d13b:da08:ecfe:a0e4) Quit (Ping timeout: 480 seconds)
[18:31] * vata (~vata@ has joined #ceph
[18:40] * jluis (~JL@ has joined #ceph
[18:40] * slang (~slang@ace.ops.newdream.net) Quit (Ping timeout: 480 seconds)
[18:45] * joao (~JL@ Quit (Ping timeout: 480 seconds)
[18:48] * glowell (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) has joined #ceph
[18:48] * sagelap1 (~sage@2607:f298:a:607:d13b:da08:ecfe:a0e4) Quit (Quit: Leaving.)
[18:48] * sagelap (~sage@ has joined #ceph
[18:49] * rongze (~zhu@ Quit (Quit: Ex-Chat)
[18:50] * loicd (~loic@ Quit (Ping timeout: 480 seconds)
[18:51] * slang (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) has joined #ceph
[18:52] <fmarchand> slang
[18:53] <fmarchand> did you find something ?
[18:53] * MikeMcClurg (~mike@firewall.ctxuk.citrix.com) has joined #ceph
[18:53] <slang> fmarchand: not yet
[18:54] * Leseb (~Leseb@ Quit (Quit: Leseb)
[18:57] * chutzpah (~chutz@ has joined #ceph
[19:03] * bchrisman (~Adium@ has joined #ceph
[19:04] * yehudasa (~yehudasa@2607:f298:a:607:a532:18b2:1f59:f68c) Quit (Quit: Ex-Chat)
[19:06] * sjustlaptop (~sam@68-119-138-53.dhcp.ahvl.nc.charter.com) has joined #ceph
[19:07] * sagelap (~sage@ Quit (Ping timeout: 480 seconds)
[19:10] * yehudasa (~yehudasa@2607:f298:a:607:ac9c:6541:7b1f:96e1) has joined #ceph
[19:12] * noob2 (a5a00214@ircip3.mibbit.com) Quit (Quit: http://www.mibbit.com ajax IRC Client)
[19:17] * Leseb (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) has joined #ceph
[19:19] * sagelap (~sage@2607:f298:a:607:d13b:da08:ecfe:a0e4) has joined #ceph
[19:20] * James259 (~James259@ Quit (Ping timeout: 480 seconds)
[19:21] * jjgalvez (~jjgalvez@cpe-76-175-17-226.socal.res.rr.com) has joined #ceph
[19:26] * yehudasa_ (~yehudasa@ has joined #ceph
[19:31] * s3_testing (~s3_testin@ipc-hosting.de) has joined #ceph
[19:33] * loicd (~loic@magenta.dachary.org) has joined #ceph
[19:34] <s3_testing> hey guys! I have a problem with rados and a S3 client. I get the error message that it failed to create auth header: NOTICE: failed to parse date for auth header. Can some one give me a hint where to look?
[19:35] * adjohn (~adjohn@ has joined #ceph
[19:42] * tryggvil_ (~tryggvil@16-80-126-149.ftth.simafelagid.is) has joined #ceph
[19:42] * tryggvil (~tryggvil@16-80-126-149.ftth.simafelagid.is) Quit (Read error: Connection reset by peer)
[19:42] * tryggvil_ is now known as tryggvil
[19:46] * guigouz (~guigouz@ has joined #ceph
[19:47] * Psi-jack (~psi-jack@psi-jack.user.oftc.net) has joined #ceph
[19:47] <Psi-jack> How is ceph doing these days, I'm curious. ;)
[19:54] <jluis> sjust, around?
[19:54] * jluis is now known as joao
[19:54] <sjustlaptop> jup
[19:54] <sjustlaptop> *yup
[19:54] <sjustlaptop> sure
[19:58] <s3_testing> so any ideas guys? :)
[20:01] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[20:01] * dmick (~dmick@2607:f298:a:607:1a03:73ff:fedd:c856) has joined #ceph
[20:02] * loicd (~loic@magenta.dachary.org) has joined #ceph
[20:02] * BManojlovic (~steki@ has joined #ceph
[20:05] <yehudasa> s3_testing: what does the HTTP_DATE show?
[20:05] <s3_testing> let me check
[20:06] <s3_testing> HTTP_DATE=Wed, 14 Nov 2012 18:34:32
[20:06] <s3_testing> but that request is rather old shall I try now?
[20:06] <s3_testing> HTTP_DATE=Wed, 14 Nov 2012 19:06:27
[20:08] <s3_testing> yehudasa: seems to be the correct time/date
[20:10] <Psi-jack> Hmmm
[20:10] <Psi-jack> trying to get ceph working, but so far, failing. Getting mount error 5 = Input/output error.
[20:11] <yehudasa> s3_testing: I'm not sure, but it looks like the wrong format
[20:12] <s3_testing> yehudasa: what do you mean by wrong format? I just installed ntpd to get the current time and synced so the time/date on the machine should be ok. What kind of format should it be? and how do I set it?
[20:13] <yehudasa> s3_testing: what client are you using?
[20:13] <s3_testing> Arq
[20:13] <s3_testing> yehudasa: what is strange though is that s3curl works fine and can create buckets
[20:14] <yehudasa> s3_testing: how does the date format with s3curl look like?
[20:14] <s3_testing> yehudasa: s3curl: HTTP_DATE=Wed, 14 Nov 2012 19:14:35 +0000
[20:15] <s3_testing> yehudasa: Arq: HTTP_DATE=Wed, 14 Nov 2012 19:15:04 GMT+00:00
[20:16] <yehudasa> earlier you didn't mention the GMT+00:00
[20:16] <s3_testing> yehudasa: sorry! I didn't paste the whole line … sorry
[20:16] <s3_testing> yehudasa: now I did :)
[20:17] <yehudasa> in any case, that's the problem
[20:17] <s3_testing> so its the client? or can I change something on the server side?
[20:17] * miroslav (~miroslav@wlan-clients-8094.sc12.org) has joined #ceph
[20:17] <yehudasa> s3_testing: are you compiling from source?
[20:17] <s3_testing> yehudasa: no I installed the ubuntu package
[20:19] <dmick> Psi-jack: so, any details at all?...
[20:19] <Psi-jack> dmick: That's.. Literally all I got. So far.
[20:19] <yehudasa> s3_testing: not much without modifying the source.. what version are you running?
[20:20] <dmick> what's your cluster like; what's your ceph.conf; what command(s) did you try; have you tried ceph -s, ceph -w; have you looked at logs...
[20:20] <s3_testing> yehudasa: ceph version 0.48.2argonaut (commit:3e02b2fad88c2a95d9c0c86878f10d1beb780bfe)
[20:23] * buck (~buck@bender.soe.ucsc.edu) has joined #ceph
[20:23] <yehudasa> s3_testing: we can relax the date check there
[20:24] <yehudasa> s3_testing: I can push it to some branch and you could pick it from the gitbuilder
[20:24] <s3_testing> yehudasa: sounds great!
[20:30] * MikeMcClurg (~mike@firewall.ctxuk.citrix.com) Quit (Quit: Leaving.)
[20:33] <yehudasa> s3_testing: are you running precise (ubuntu 12.04)?
[20:33] <s3_testing> yes
[20:34] <joshd> fmarchand: is your ceph compiled against tcmalloc (in google-perftools)? if not, it'll use much more memory
[20:34] <tnt> Dah ... somehow firefox sends a 'content-type' header even when doing a HEAD and that screws with token auth of RadosGW ...
[20:35] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has joined #ceph
[20:35] <sagewk> slang: where are we at with wip-3431?
[20:35] <sagewk> ready to merge, right?
[20:36] <slang> sagewk: sorry - yes, ready for merge
[20:36] <yehudasa> s3_testing: add 'deb http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-basic/ref/wip-rgw-relaxdate/ precise main' to your apt sources list
[20:36] <yehudasa> s3_testing: it's not ready yet though
[20:37] <yehudasa> s3_testing: once http://ceph.com/gitbuilder.cgi shows it in green
[20:37] <s3_testing> yehudasa: ok, adding it now and waiting for green :)
[20:37] <yehudasa> then do sudo apt-get update; sudo apt-get install radosgw
[20:38] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has left #ceph
[20:38] <s3_testing> yehudasa: ok will do! so its radosgw which we update …
[20:38] <yehudasa> yeah
[20:38] <Psi-jack> Ahh
[20:39] <Psi-jack> I figured it out. The problem was is the hostname and IP differ in meanings. Because the fileserver I was putting ceph on is dual-homed, LAN Network and Storage Network separated.
[20:40] <Psi-jack> I'm also understanding that ceph is storing it's data into /var/lib/ceph/* ?
[20:40] * gaveen (~gaveen@ Quit (Remote host closed the connection)
[20:42] <dmick> Psi-jack: by default, much of the data goes there, yes
[20:43] <dmick> all configurable of course
[20:43] <s3_testing> yehudasa: is there a specific one which needs to be green or just all of them?
[20:48] * Psi-jack nods.
[20:48] <Psi-jack> So, now all I need to do is figure out how to get ceph to work on my storage network, instead of my normal network. ;)
[20:48] <Psi-jack> Yeaaah, it's my secondary storage server not being able to communicate with the storage network. :/
[20:50] <Psi-jack> Heh, ceph was pumping so much bandwidth through my local network that it squelched access to my webservers. LOL
[21:01] <dmick> Psi-jack: see http://ceph.com/docs/master/config-cluster/ceph-conf/#networks
[21:06] <exec> folks. after some data migration/cluster resize I've got problem with one of rbds:
[21:06] <exec> rbd ls yyy
[21:06] <exec> error opening pool yyy: (2) No such file or directory
[21:06] <exec> rbd info yyy
[21:06] <exec> error opening image yyy: (2) No such file or directory
[21:06] <exec> 2012-11-14 19:54:02.906632 7f94f4591760 -1 librbd: error finding header: (2) No such file or directory
[21:07] <s3_testing> yehudasa: awesome!!!!!!!!!! it works! it works :) I am working now 4 weeks in getting a S3 API implemtend to work with Arq and now it works! thanks so much
[21:07] <exec> what can I do? and where to see? it can't be mapped, attached to VM or so
[21:07] <elder> yehudasa is the bomb.
[21:07] <s3_testing> ceph is awesome :)
[21:07] <s3_testing> and yehudasa as well he he
[21:08] <s3_testing> just so I know if I update my installation I will lose this relaxation of date check right?
[21:16] * fmarchand2 (~fmarchand@ has joined #ceph
[21:16] <fmarchand2> hi !
[21:17] <fmarchand2> slang : about the zombie file maybe I found something ...
[21:18] <slang> fmarchand2: oh?
[21:18] <fmarchand2> I checked the version of ceph-fs-tools on the machine
[21:20] * mdrnstm (~mdrnstm@206-169-78-213.static.twtelecom.net) has joined #ceph
[21:20] <fmarchand2> and it is a 0.34 ... should it be a 0.48 like the ceph version of the cluster ?
[21:22] <yehudasa> s3_testing, elder: thanks
[21:23] <yehudasa> s3_testing: we'll push that to argonaut, and to bobtail, so that future update will have that
[21:24] <Psi-jack> dmick: Cool, will check it out.
[21:28] <s3_testing> yehudasa: awesome thanks so much again!!! :) you made my day
[21:32] <Psi-jack> dmick: Now, how come, when I change the network settings in /etc/ceph/ceph.conf and start ceph services back up, it's still binding to the old IP previously setup?
[21:33] <joshd> exec: does 'rados -p rbd ls | grep yyy' show anything?
[21:33] <joshd> exec: the header object is named after the image
[21:34] <joshd> exec: it'll show up as a file with the object's name in it on the osds, if you want to check on that level
[21:35] <dmick> Psi-jack: probably because the monitor map was built with the old addresses
[21:35] <slang> fmarchand2: the tools should match, but I doubt that's the cause of your problem
[21:35] <dmick> (presuming you mean ceph-mon processes)
[21:36] <fmarchand2> I'm gonna try to install a newer package anyway I'll tell you
[21:36] <Psi-jack> dmick: There any way to "update" that?
[21:37] <dmick> Psi-jack: maybe, but I'm not sure how hard it is
[21:37] <dmick> might be easier to re-init the cluster if you don't have any data you care about yet
[21:37] <fmarchand2> I just hope it's not a configuration pb on my side ... I don't like to waste time of somebody ! I'm sure you're busy !
[21:38] <Psi-jack> dmick: Yeaaah.. Heh
[21:38] <Psi-jack> Nothing important... Yet.
[21:38] <dmick> Psi-jack: how did you initially deploy the cluster (with which tools)?
[21:39] <Psi-jack> dmick: I just followed the quick start doc: http://ceph.com/docs/master/start/quick-start/
[21:40] <dmick> so, mkcephfs
[21:42] <dmick> yeah, that would have created a monmap with monmaptool and distributed it. I've never tried to update a monmap.
[21:42] * adjohn (~adjohn@ Quit (Quit: adjohn)
[21:42] * miroslav (~miroslav@wlan-clients-8094.sc12.org) Quit (Quit: Leaving.)
[21:42] <dmick> http://ceph.com/docs/master/cluster-ops/add-or-rm-mons/ is probably useful info if you want to try
[21:43] * adjohn (~adjohn@ has joined #ceph
[21:45] * styx__ is now known as styx
[21:46] <fmarchand2> slang : ok ... you were right I still can't remove my folder with the ceph-fs-common v0.48.2 argonaut
[21:47] <fmarchand2> slang : did you ever heard about a bug like mine or I'm the only one who met that issue ?
[21:48] <Psi-jack> dmick: Heh, might use that guide to add my secondary storage server to the mix.:)
[21:48] <slang> fmarchand2: there was just someone on the mailing list that saw a similar issue
[21:48] <dmick> Psi-jack: keep in mind that number and location of monitors is independent of number and location of OSDs
[21:49] <fmarchand2> slang : I sent it on the mailing list about 2 days ago... maybe it was me ?
[21:49] * LarsFronius (~LarsFroni@testing78.jimdo-server.com) Quit (Ping timeout: 480 seconds)
[21:50] <slang> fmarchand2: ah yep
[21:54] <fmarchand2> slang : I know that when you're the only one having a pb .... it's probably not the application fault but the way you use it ... :)
[21:55] <slang> fmarchand2: it looks like its a bug that only gets triggered very infrequently
[21:55] <slang> fmarchand2: and unfortunately to reproduce it we need the logs when the error actually occurred
[21:56] <fmarchand2> slang : and the logs I sent you had some error lines ?
[21:58] * miroslav (~miroslav@wlan-clients-8094.sc12.org) has joined #ceph
[22:03] <slang> fmarchand2: which kernel version are you running?
[22:03] * adjohn (~adjohn@ Quit (Quit: adjohn)
[22:04] * s3_testing (~s3_testin@ipc-hosting.de) Quit (Quit: s3_testing)
[22:04] * MikeMcClurg (~mike@cpc10-cmbg15-2-0-cust205.5-4.cable.virginmedia.com) has joined #ceph
[22:05] <fmarchand2> slang : on the client machine or the cluster machine ?
[22:06] <slang> client
[22:07] <fmarchand2> 3.0.0-12-generic
[22:08] <fmarchand2> slang : for the client there is no kernel compilation to do or whatever ?
[22:09] * LarsFronius (~LarsFroni@2a02:8108:380:12:dcb9:8663:3b6f:d0c7) has joined #ceph
[22:16] * MikeMcClurg (~mike@cpc10-cmbg15-2-0-cust205.5-4.cable.virginmedia.com) Quit (Quit: Leaving.)
[22:17] * MikeMcClurg (~mike@cpc10-cmbg15-2-0-cust205.5-4.cable.virginmedia.com) has joined #ceph
[22:22] <Psi-jack> dmick: Yeah.. Well, I'll probably have only 2 monitors, 2 or 4 osd's, and... However many mds's are needed. I have 2 storage servers to work with.
[22:22] <dmick> you want an odd number of monitors.
[22:22] * nhorman (~nhorman@hmsreliant.think-freely.org) Quit (Quit: Leaving)
[22:22] <dmick> 2 is no better than one, because if one fails, you're stuck with no quorum
[22:23] <Psi-jack> Hmmm
[22:23] <Psi-jack> Well, the monitor doesn't need to be with the data nodes, does it?
[22:24] <Psi-jack> If not, I could throw the 3rd monitor up on one of the hypervisor hosts itself, depending on what it needs.
[22:24] <dmick> it does not
[22:24] <dmick> and it uses fairly few resources
[22:26] <Psi-jack> Cool then yeah.
[22:26] <Psi-jack> I could easily toss that into a hypervisor's host just to make up the odd number.
[22:27] <Psi-jack> It's that, or a VM within one of them, but since I plan on trying out ceph to house my VM disks... I kinda don't want a VM to depend on being available to provide quorum. ;)
[22:27] <Psi-jack> Also, does ceph work well over ZFS?
[22:28] <dmick> Psi-jack: not certain. Some have tried, and reported issues
[22:28] <dmick> they sounded minor but I don't know that anyone's plumbed the depths yet
[22:28] <dmick> maybe O_DIRECT support?...
[22:29] <dmick> http://tracker.newdream.net/issues/3440
[22:29] <dmick> http://tracker.newdream.net/issues/3287
[22:29] * yehudasa_ (~yehudasa@ Quit (Read error: Operation timed out)
[22:31] <Psi-jack> Hmm
[22:32] <Psi-jack> And that was on Ubuntu 12.04 too :(
[22:32] <dmick> we'd be happy fro help reproducing and diagnosing :)
[22:32] <Psi-jack> hehe
[22:32] <dmick> and we'll even accept patches
[22:33] <Psi-jack> I'm sure. Well, I'm not tossing anything critical on these anyway. But, I don't program C/C++, so I can only report exact details of what I see, and how to reproduce.
[22:33] <dmick> that could still be handy. At least one of those bugs was waiting for more input from the submitter
[22:34] * Psi-jack nods.
[22:34] <Psi-jack> ZFS also, of course, allows to make a virtual block device so you could throw on another filesystem, but you loose some of the benefits of ZFS due to that. ;)
[22:34] <benpol> Just hit the enter key prematurely (typed: "rbd resize foo"), expected an error, but now I'm seeing "Resizing image..." progress output. Given that I didn't specify a new size, what's actually happening?
[22:35] <dmick> benpol: a bug. hm. looking at source
[22:36] <dmick> *looks* like it'll resize to 0?...
[22:36] <benpol> dmick: wouldn't that be interesting.
[22:36] <joshd> yeah, I'm pretty sure that's right dmick
[22:36] <dmick> indeed
[22:36] <dmick> $ ./rbd ls -l
[22:36] <dmick> image 1024K 1
[22:36] <dmick> $ ./rbd resize image
[22:36] <dmick> Resizing image: 100% complete...done.
[22:37] <joshd> the trouble with not having a real argument parsing framework
[22:37] <dmick> $ ./rbd ls -l
[22:37] <dmick> image 0 1
[22:37] <benpol> joshd: what you said!
[22:37] <Psi-jack> dmick: Heh, my testing platform is rather interesting. Basically trying to see if I can setup Proxmox VE 2.2 to work with Ceph, manually.
[22:39] <Psi-jack> So, in ceph.conf, osd journal = /blah/$name/journal is where you set the journal? How do you setup where the actual data will be stored?
[22:39] <benpol> joshd: any movement towards a comprehensive argument parsing framework?
[22:39] <Psi-jack> I'll start it off just by providing a ZFS subvolume for /ceph for the two main monitors, and an ext4 volume from one of the hypervisors.
[22:40] <dmick> http://ceph.com/docs/master/config-cluster/ will come in handy
[22:40] <joshd> benpol: some benchmarking tools we have started using boost program options. I'm not sure how much better it would be
[22:41] <dmick> for that specifically, "osd data"
[22:41] <Psi-jack> Ahhh
[22:43] <Psi-jack> And lastly, is there a reason I should have the ceph servers setup to have public network and cluster network? The docs you referred me to on that aren't really clear how they work entirely, and my 2 storage servers plus 4 hypervisors all are connected directly via the cluster network.
[22:44] * adjohn (~adjohn@ has joined #ceph
[22:45] <lurbs> Psi-jack: My Ceph servers have separate networks for the storage and admin. I found it useful for client (like, say, OpenStack) to be able to access them without being directly on the storage network.
[22:46] <lurbs> s/client/clients/
[22:46] <lurbs> Plus the storage is a separate 10 Gb infrastructure, which gets expensive real quick.
[22:46] <Psi-jack> lurbs: I see. Mine will all be able to access them over the storage network as my hypervisors will be directly attached to the physical network. I don't want ceph traversing over my infrastructure network as I've already tested that out and watched it quickly drain my network to high latency. ;)
[22:47] <dmick> Psi-jack: you can use one net. If you want to separate, the 'back' network cluster traffic can be segregated for efficiency (osd-to-osd traffic, like replication/recovery)
[22:47] * Steki (~steki@ has joined #ceph
[22:48] <benpol> dmick: confirmed... image is now size 0 (glad I had a snapshot)
[22:48] <Psi-jack> Could I actually set osd data = /ceph/osd/$cluster-$id or do I need to replace $cluster and $id with actual values?
[22:48] * BManojlovic (~steki@ Quit (Read error: Connection reset by peer)
[22:49] * fmarchand2 (~fmarchand@ Quit (Ping timeout: 480 seconds)
[22:50] <joshd> Psi-jack: that will work
[22:50] <lurbs> $cluster defaults to 'ceph', I think, unless you do something specific in the cluster setup. And $id the the id of each osd/mon/mds, so you can use the variable.
[22:50] <Psi-jack> Cool. ;)
[22:50] <lurbs> Someone will let me know if I'm telling filthy lies, right?
[22:51] <Psi-jack> So, I have 2 storage servers. I'll need 2 osd's, 2 mds's, and 3 mon's?
[22:51] <joshd> lurbs: that's right
[22:52] <joshd> Psi-jack: mds's are only needed for cephfs, and don't store anything locally
[22:53] <Psi-jack> Hmmm
[22:54] <iggy> and the kernel rbd client can't be used on osd's
[22:57] * Leseb_ (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) has joined #ceph
[22:57] * Leseb (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) Quit (Read error: Connection reset by peer)
[22:57] * Leseb_ is now known as Leseb
[23:00] <yehudasa> elder / scuttlemonkey: can you change the channel's tag line
[23:00] <yehudasa> ?
[23:01] <elder> You want it to say, "yehudasa is the bomb!"?
[23:01] <yehudasa> elder: I'd rather, much too humble for that
[23:01] <yehudasa> but we can have it saying 0.54 is out
[23:02] <yehudasa> I'd rather *not*
[23:02] <elder> I know. Let me see what I can do.
[23:03] * senner (~Wildcard@68-113-232-90.dhcp.stpt.wi.charter.com) Quit (Quit: Leaving.)
[23:03] * fmarchand2 (~fmarchand@ has joined #ceph
[23:04] <scuttlemonkey> you want just that, or include the github and/or tarball links?
[23:05] * vjarjadian (~IceChat7@5ad6d001.bb.sky.com) has joined #ceph
[23:05] * scuttlemonkey changes topic to 'v0.54 is out'
[23:05] <vjarjadian> hi
[23:06] <elder> Damn!
[23:06] <scuttlemonkey> =D
[23:06] <Psi-jack> Does the mon use much disk space?
[23:06] <elder> I just figured out how to do that!
[23:06] <scuttlemonkey> hehe
[23:07] <buck> autobuild-ceph question. I'm adding a builder for Hadoop, but the code is setup to use the same branches-local script for everything. There's a lot of broilerplate code around the call to branches-local that I'm loathe to copy/paste. I'm planning on parameterizing the name of the branches-local script in the _gitbuilder() call. Does that seem reasonable?
[23:07] * elder changes topic to 'v0.54 is released'
[23:07] <elder> :)
[23:08] <scuttlemonkey> hah
[23:09] <scuttlemonkey> actually, since Sage did such a nice blog entry w/ all the links, we should just use that
[23:10] * scuttlemonkey changes topic to 'v0.54 has been released -- http://goo.gl/vgRR1'
[23:10] <joshd> Psi-jack: not too much, check out http://ceph.com/docs/master/install/hardware-recommendations/#minimum-hardware-recommendations
[23:11] <benpol> so after my accidental resize to zero operation, I thought I'd try an rbd snap rollback, but "rbd snap rollback rbd/foo@20121112-162046" returns "Rolling back to snapshot: 0% complete...failed. rbd: rollback failed: (2) No such file or directory"
[23:11] <Psi-jack> Hmm, 10GB per mon, eh? A bit more than I expected, but okay. ;)
[23:11] <benpol> fortunately I seem to be able to clone the snapshot to a new volume (and am able to flatten it too).
[23:14] <elder> dmick, is rbd CLI able to specify an image that's something that's not a multiple of 1MB in size?
[23:14] <elder> Or joshd
[23:14] <joshd> elder: yes
[23:14] <elder> Can you tell me how?
[23:15] <elder> Ah , rbd -h
[23:15] <elder> (not rbd help, not rbd ?)
[23:15] <elder> No help after all. How?
[23:15] <elder> It says "create --size <MB>
[23:15] <joshd> --order
[23:16] <elder> That would be worse I think.
[23:16] <elder> I'm trying to avoid some problems XFS is reporting relating to things lining up just too perfectly.
[23:17] <joshd> there's no way to make it not 4k-aligned from the rbd cli tool
[23:17] <iggy> similar to the ext4 unaligned directio warning?
[23:17] <elder> Having an image with a different size might be an easy fix. I'll see if I can configure some XFS parameters to get around it.
[23:18] <elder> 4K aligned would be great. But it looks like I'm stuck with 2^20 aligned.
[23:18] <joshd> elder: the min and opt sizes reported are base on object size though
[23:18] <elder> Right.
[23:18] <elder> That's not the issue.
[23:18] <iggy> are we talking rbd?
[23:18] <joshd> elder: --order 12 isn't enough?
[23:18] <elder> iggy yes.
[23:18] <elder> --order 12 will give me what?
[23:18] <joshd> 4k object size
[23:19] <elder> I'm talking about the *image* size.
[23:19] <iggy> are the guest partitions aligned to 1M (or similar)
[23:19] <joshd> ah, ok. the cli only does MB sizes
[23:19] <iggy> most installers still misalign things
[23:20] <elder> I'm formatting a single file system on a single rbd image. XFS breaks the available space up into allocation groups.
[23:20] <iggy> so unpartitioned?
[23:20] <elder> Based on the sizes it's getting from probing the rbd "hardware" it's coming up with allocation groups that it thinks are too well aligned.
[23:20] <elder> So it's reporting a warning. That's fine, but I'm trying to get my tests to run without any warnings.
[23:20] <elder> iggy, yes.
[23:20] <iggy> try playing with stride size on the mkfs
[23:21] <Psi-jack> heh.
[23:21] <Psi-jack> Well now... mkcephfs already failed to work on ZFS. ;)
[23:21] <benpol> So not only will "rbd resize foo" result in an image with a size of 0, it seems to break "rbd snap rollback" functionality. That's a potentially painful bug.
[23:21] <elder> I can set my agsize or agcount and I think that should take care of it.
[23:21] <Psi-jack> 7f67b9153780 -1 OSD::mkfs: FileStore::mkfs failed with error -22
[23:21] <Psi-jack> 2012-11-14 17:20:59.775294 7f67b9153780 -1 ** ERROR: error creating empty object store in /ceph/osd/ceph-0: (22) Invalid argument
[23:23] * Steki (~steki@ Quit (Remote host closed the connection)
[23:23] <dmick> Psi-jack: yes, I think that's the O_DIRECT thing
[23:23] <fmarchand2> A quick question : what is scrubbing for an osd ?
[23:24] <joshd> benpol: I suspect there's something strange going on, snapping, resizing to zero, and rolling back works fine for me
[23:24] <benpol> joshd: that could be (though I was able to successfully do a rollback of a different rbd image)
[23:25] <dmick> journal dio = false will make it not do that
[23:25] <dmick> https://github.com/zfsonlinux/zfs/issues/224
[23:28] <benpol> joshd: Interesting, seems it was a transient error, my rollback of the zero'ed out image seems to be working now.
[23:28] * sjustlaptop (~sam@68-119-138-53.dhcp.ahvl.nc.charter.com) Quit (Quit: Leaving.)
[23:29] * sjustlaptop (~sam@68-119-138-53.dhcp.ahvl.nc.charter.com) has joined #ceph
[23:30] * sjustlaptop (~sam@68-119-138-53.dhcp.ahvl.nc.charter.com) Quit ()
[23:30] <joshd> benpol: if it ever reproduces, adding --debug-ms 1 and --debug-rbd 20 would be useful
[23:31] <benpol> joshd: making a note of those flags, thanks
[23:33] * sjustlaptop (~sam@68-119-138-53.dhcp.ahvl.nc.charter.com) has joined #ceph
[23:34] <Psi-jack> dmick: Yeah, when I created a zvol block device and made it ext4, worked.. LOL
[23:35] <Psi-jack> Hmmm
[23:35] <Psi-jack> Or... maybe...
[23:44] * dspano (~dspano@rrcs-24-103-221-202.nys.biz.rr.com) Quit (Quit: Leaving)
[23:45] <jlogan1> Can someone help me get libvirt with a rbd pool working?
[23:48] <joshd> jlogan1: are you trying to use libvirt pools, or just attach rbd images to vms?
[23:49] <jlogan1> I was trying to use the pool, so foreman could make the VMs
[23:49] <jlogan1> I can reproduce the error with virsh commands
[23:49] <jlogan1> root@ceph01-sef:/etc/libvirt/storage# virsh vol-create-as data test7-sef.prod.msrch-disk1 20971520
[23:49] <jlogan1> error: Failed to create vol test7-sef.prod.msrch-disk1
[23:49] <jlogan1> error: internal error failed to create volume 'data/test7-sef.prod.msrch-disk1'
[23:49] <jlogan1> 2012-11-14 22:48:47.844+0000: 7202: error : virSecretFree:15443 : Invalid secret: virSecretFree
[23:49] <jlogan1> 2012-11-14 22:48:47.846+0000: 7202: error : virStorageBackendRBDCreateVol:442 : internal error failed to create volume 'data/test7-sef.prod.msrch-disk1'
[23:52] * dmick (~dmick@2607:f298:a:607:1a03:73ff:fedd:c856) Quit (Remote host closed the connection)
[23:55] <Psi-jack> Heh, interesting. Watching bonnie++ disk usage basically split between the two osd's nearly evenly.
[23:55] <joshd> jlogan1: that sounds like a possible reference counting bug for the secret
[23:56] <joshd> jlogan1: could you increase the libvirtd log level to debug?
[23:56] <jlogan1> joshd: Should I need to setup a secret or a secret uuid for the pool? I did not do that.
[23:58] <Psi-jack> Though, I do have one issue currently plaguing me. I have Ceph installed on Debian 6 from the ceph repositories, but when I try to mount ceph, I get this: FATAL: Module ceph not found; mount.ceph: modprobe failed, exit status 1; mount error: ceph filesystem not supported by the system
[23:59] <joshd> jlogan1: you shouldn't need to if you're not using cephx (and didn't specify a rados user in the pool definition)

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.