#ceph IRC Log


IRC Log for 2012-10-26

Timestamps are in GMT/BST.

[0:01] * phl (~user@ Quit (Remote host closed the connection)
[0:04] * PerlStalker (~PerlStalk@perlstalker-1-pt.tunnel.tserv8.dal1.ipv6.he.net) Quit (Quit: rcirc on GNU Emacs 24.2.1)
[0:10] * nwatkins2 (~Adium@soenat3.cse.ucsc.edu) has joined #ceph
[0:10] * nwatkins1 (~Adium@soenat3.cse.ucsc.edu) Quit (Read error: Connection reset by peer)
[0:10] * amatter_ (~amatter@ has joined #ceph
[0:12] * amatter_ (~amatter@ Quit ()
[0:12] <buck> joshd: with that change I wanted to make to teuthology, it looks like the correct way to do it is to sort out which target is executing the given task and then pull the user from the target definition (the characters prior to the @ symbol). Sound reasonable? I'm just looking to sanity check the idea.
[0:14] <Robe> are rados questions also welcome here?
[0:14] * dspano (~dspano@rrcs-24-103-221-202.nys.biz.rr.com) Quit (Quit: Leaving)
[0:14] * amatter (~amatter@ Quit (Ping timeout: 480 seconds)
[0:14] <gregaf> rados is part of the Ceph project ;)
[0:14] <gregaf> bring it on, Robe
[0:15] <Robe> I was wondering if there's a good side-by-side comparison of rados vs. riak
[0:15] <Robe> Since everything I gathered so far pointed at pretty similar overall goals and design considerations
[0:16] <gregaf> I'm not aware of any, no
[0:16] <gregaf> RADOS isn't trying to be a database of any kind though (NoSQL or otherwise); it's a pure object store
[0:17] <Robe> well.
[0:17] <Robe> it has an API, so... ;)
[0:17] <gregaf> unlike something like Dynamo, RADOS is a strictly-consistent system
[0:17] * glowell (~Adium@c-98-210-224-250.hsd1.ca.comcast.net) has joined #ceph
[0:17] <Robe> no eventuality?
[0:17] <gregaf> nope
[0:18] <gregaf> I'm not really familiar enough with Riak to offer anything else
[0:18] <joshd> buck: yeah, and you can get that from the remote the command is being executed with
[0:19] <Robe> gregaf: well, with Riak every client decides his level of integrity (amount of replicas) for write and read operations
[0:19] <buck> joshd: I'm not entirely clear on what you mean when you say "the remote". Could you clarify.
[0:19] <Robe> and concurrent writes are handled strictly on application level
[0:20] <gregaf> Robe: yes, that's the eventually-consistent model
[0:20] <Robe> how does Rados approach this?
[0:20] <Robe> without violating any laws of physics? ;)
[0:20] <gregaf> RADOS is entirely strict consistency; you can have data with different numbers of copies by placing it in different "pools"
[0:21] <Robe> and every put/update only returns to the client after the consistency level has been reached?
[0:21] <gregaf> it does a lot more accounting work to try and minimize any availibility issues, but it chooses consistency over availability
[0:21] <gregaf> correct
[0:21] <Robe> kinky.
[0:21] <joshd> buck: I mean the variable named remote in _make_scratch_dir - it's of type orchestra.remote.Remote
[0:22] <joshd> buck: its name attribute is the user@host for making the ssh connection
[0:22] <Robe> this on the other hand means that lazy/failing osds can nuke the entire cluster till they're proper dead
[0:22] <gregaf> Robe: basically, yes
[0:22] <buck> joshd: oh rad. I thought I'd have to suss all that info out of the ctx myself. Thanks for the info and the pointer to that variable
[0:23] <Robe> gregaf: is there a good design overview for rados?
[0:23] <joshd> buck: no problem, let me know if you have other teuthology questions
[0:23] * aliguori (~anthony@ Quit (Quit: Ex-Chat)
[0:23] <gregaf> it's not super likely in that the OSDs keep track of each other and they'll get marked down within (by default, this is tunable) 30-40 seconds of dying, and they also keep track of themselves and suicide if for instance their disks stop responding
[0:23] <dmick> buck: and sorry for the mistaken review. I misunderstood the point of your change and just assumed too much
[0:24] <gregaf> well, you can read the academic papers
[0:24] <Robe> I've seen the CRUSH paper
[0:24] <Robe> not too sure if it covers the rest of the aspects
[0:24] <gregaf> there's also a RADOS paper
[0:24] <gregaf> rturk: scuttlemonkey_: can one of you take this?
[0:24] <gregaf> gotta get on a phone call
[0:25] <Robe> haha
[0:25] <Robe> no worries :)
[0:25] <Robe> http://ceph.com/papers/weil-rados-pdsw07.pdf that one?
[0:26] <joshd> yeah, that's the one
[0:26] <Robe> great, thanks
[0:26] <buck> dmick: you assumed I did the right thing and reviewed the code in that light. I should have verified that I was taking the right tack in the first place.
[0:26] <joshd> there's also sage's whole thesis if you want a lot of detail
[0:26] * vata (~vata@ Quit (Quit: Leaving.)
[0:27] <buck> somewhere a whole forest just shrieked in terror of a thesis being printed
[0:27] <Robe> is it public as well?
[0:27] <Robe> buck: I've got an ipad, no worries ;)
[0:30] <joshd> Robe: yeah, same place: http://ceph.com/papers/weil-thesis.pdf
[0:34] <Robe> great, thanks :)
[0:35] <Robe> and on a side note - you've been one of the most helpful IRC communities I've encountered so far
[0:36] * BManojlovic (~steki@ Quit (Quit: Ja odoh a vi sta 'ocete...)
[0:37] <rweeks> we aim to please!
[0:38] <Robe> AAA+ Would pester again!
[0:38] <Robe> ;)
[0:40] * Q (~qgrasso@ip-121-0-1-110.static.dsl.onqcomms.net) has joined #ceph
[0:41] * Q is now known as Guest3170
[0:43] <Robe> holy moly
[0:43] <Robe> that thesis is some good filesystem porn
[0:44] <gregaf> heh
[0:44] <gregaf> fyi, it talks about three replication strategies but the only one used any more is primary replication, so if you actually read that far skip the other ones
[0:44] <nwatkins2> sagewk: i'm noticing really long mount times again, (~60 seconds), with subsequent mounts by the same process being fast. last time the fix was something to do with renewing caps. could some of the mds stuff recently merged have brought this back?
[0:45] <Robe> first bed
[0:45] <Robe> and probably will be in amsterdam next week
[0:45] <gregaf> going to the Ceph day event?
[0:46] <Robe> yesh
[0:46] <gregaf> that's probably the easiest way to figure stuff out
[0:46] <Robe> if on42 allows me to participate
[0:46] <Robe> paypal apparently haets my creditcard
[0:46] <gregaf> ah, heh
[0:46] <gregaf> I'll look forward to seeing you, then :)
[0:47] <Robe> likewise!
[0:47] <rweeks> paypal across country borders is impossible.
[0:47] <mikeryan> rweeks: just use bitcoin!
[0:47] <Robe> hrm!
[0:47] * rweeks snorts
[0:47] <Robe> that could be it
[0:47] <Robe> currently in prague
[0:47] <rweeks> yeah, paypal and national borders = bad thing
[0:48] <rweeks> I don't know what it's like in Europe, but even from US to Canada it sucks
[0:48] <Robe> it's time klarna gets their speed up
[0:48] <Robe> rweeks: well... at least operating from inside austria to the rest of the world mostly worked with paypal
[0:48] <Robe> though I try to avoidt it where I can
[0:50] <rturk> Robe: I'll be at the ceph day
[0:50] <rturk> sorry, was away earlier when greg had a phone call :)
[0:51] <Robe> I found your lack of presence... disturbing!
[0:51] <Robe> just kidding ;)
[0:51] * rweeks snorts
[1:09] * tryggvil (~tryggvil@163-60-19-178.xdsl.simafelagid.is) has joined #ceph
[1:11] * yoshi (~yoshi@p37219-ipngn1701marunouchi.tokyo.ocn.ne.jp) has joined #ceph
[1:12] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[1:13] * cdblack (86868b48@ircip3.mibbit.com) Quit (Quit: http://www.mibbit.com ajax IRC Client)
[1:17] * lofejndif (~lsqavnbok@82VAAHGCV.tor-irc.dnsbl.oftc.net) Quit (Quit: gone)
[1:21] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[1:36] * LarsFronius (~LarsFroni@95-91-242-161-dynip.superkabel.de) Quit (Quit: LarsFronius)
[1:37] * Leseb (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) Quit (Quit: Leseb)
[1:57] * miroslav (~miroslav@c-98-248-210-170.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[2:00] * sjustlaptop (~sam@ has joined #ceph
[2:01] * anna1 (~Adium@c-98-234-186-68.hsd1.ca.comcast.net) has joined #ceph
[2:10] * sjustlaptop (~sam@ Quit (Ping timeout: 480 seconds)
[2:14] * rweeks (~rweeks@ Quit (Quit: ["Textual IRC Client: www.textualapp.com"])
[2:18] * anna1 (~Adium@c-98-234-186-68.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[2:32] * sagelap (~sage@2607:f298:a:607:aca0:7479:696a:6818) Quit (Ping timeout: 480 seconds)
[2:33] * sagelap (~sage@186.sub-70-197-146.myvzw.com) has joined #ceph
[2:35] * nwatkins2 (~Adium@soenat3.cse.ucsc.edu) Quit (Quit: Leaving.)
[2:36] * noob2 (a5a00214@ircip2.mibbit.com) Quit (Quit: http://www.mibbit.com ajax IRC Client)
[2:37] * Cube2 (~Cube@ has joined #ceph
[2:38] * dmick (~dmick@ Quit (Quit: Leaving.)
[2:40] * Cube1 (~Cube@2607:f298:a:697:20f5:750e:4dc9:c91c) Quit (Read error: Connection reset by peer)
[2:50] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) Quit (Quit: Leaving.)
[2:53] * buck (~buck@bender.soe.ucsc.edu) has left #ceph
[3:09] * Leseb (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) has joined #ceph
[3:10] * Leseb (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) Quit ()
[3:13] <kavonr> yay, degraded percentage is falling still.
[3:15] <kavonr> If I'm wanting to mess with cephfs, should I build from source? I started out with the apt repo, but it seems to be a ways behind.
[3:17] <joshd> kavonr: you can get updated packages http://ceph.com/docs/master/install/debian/#add-development-release-packages
[3:19] <kavonr> ah, dev release, cool.
[3:19] <kavonr> Thanks.
[3:20] <kavonr> is the upgrade pretty transparent from 0.48? I didn't find anything on that in my (admittedly) brief search for upgrade details
[3:20] <joshd> yeah, if it's not, it's a bug
[3:21] <joshd> just upgrade and restart the daemons
[3:22] <kavonr> one more question for now. :) If I'm still showing degraded from 'ceph -s' and 'ceph -w', should I wait for that to go away before upgrading, or will it pick up the recovery cleanly after upgrade?
[3:23] <joshd> it should pick up cleanly. there is one regression in 0.53 (which is the latest dev release) that's since been fixed - if you're using a block device for a journal, you need to set 'osd journal size = 0' or it won't use the whole block device
[3:24] <kavonr> ok. I'm using a file in the osd.# directory at the moment so that should be okay.
[3:26] * sagelap (~sage@186.sub-70-197-146.myvzw.com) Quit (Ping timeout: 480 seconds)
[3:29] * dmick (~dmick@ has joined #ceph
[3:36] * adjohn (~adjohn@ Quit (Quit: adjohn)
[3:36] <dmick> wee, ceph builds on quantal
[3:37] <nhmlap> yay
[3:45] * justinwa1 (~Thunderbi@ Quit (Quit: justinwa1)
[3:45] * justinwarner (~Thunderbi@ has joined #ceph
[3:50] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[3:51] * justinwarner (~Thunderbi@ Quit (Quit: justinwarner)
[3:51] * justinwarner (~Thunderbi@ has joined #ceph
[3:52] * loicd (~loic@magenta.dachary.org) has joined #ceph
[3:59] * slang (~slang@173-163-208-195-westflorida.hfc.comcastbusiness.net) has joined #ceph
[4:07] * slang (~slang@173-163-208-195-westflorida.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[4:13] * slang (~slang@173-163-208-195-westflorida.hfc.comcastbusiness.net) has joined #ceph
[4:14] * nhmlap (~nhm@ Quit (Ping timeout: 480 seconds)
[4:23] * nhmlap (~nhm@ has joined #ceph
[4:32] * rweeks (~rweeks@ has joined #ceph
[4:32] <kavonr> hi rweeks
[4:32] <rweeks> hallo
[4:33] <kavonr> < @gallifreyan
[4:33] <rweeks> ah!
[4:33] <rweeks> how goes it?
[4:33] <kavonr> not bad, brought two more nodes up today, updated to testing, updated kernels and stuff. now trying to put some more monitor daemons up
[4:34] <kavonr> 11.084% degraded but that's better than most of the day. :)
[4:34] <rweeks> cool
[4:34] <kavonr> not bad for 24 hours in the technology.
[4:35] <rweeks> no indeed
[4:39] <rweeks> so no major problems?
[4:39] <kavonr> it's been a bit confusing figuring out how to add additional servers, found some bad links in wikis and stuff... and cephfs was really bogging down when I was 25% degraded.
[4:40] * chutzpah (~chutz@ Quit (Quit: Leaving)
[4:40] <kavonr> but it's been easier than I expected, and works pretty well so far
[4:40] <kavonr> I promised Gina I'd give it a shot anyway, and a good use came up recently. :)
[4:40] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[4:41] <rweeks> I don't know if our tech pubs guy is in here, but he reports to rturk
[4:41] * sjustlaptop (~sam@ has joined #ceph
[4:41] <kavonr> I'm going to go through it again later, take better notes (i.e. take notes), and probably write a blog post.
[4:43] * loicd (~loic@magenta.dachary.org) has joined #ceph
[4:45] <kavonr> osdmap e1170: 24 osds: 24 up, 24 in
[4:46] <rweeks> what's the use case you're looking at?
[4:47] <kavonr> right now, aggregating storage on 8x1tb 1u boxes with replication.
[4:48] <rweeks> so unstructured data in filesystem?
[4:49] <kavonr> yup
[4:50] * adjohn (~adjohn@108-225-130-229.lightspeed.sntcca.sbcglobal.net) has joined #ceph
[4:50] <kavonr> specifically backing up a set of backup directories, millions of files, then dropping them back out onto a DR-type environment when that environment is built
[4:51] <rweeks> gotcha
[4:52] * sjustlaptop (~sam@ Quit (Ping timeout: 480 seconds)
[4:54] <rweeks> gotta run, might be back in a while
[4:55] <kavonr> me too, I smell dinner.
[4:55] <rweeks> hehe
[4:55] * rweeks (~rweeks@ Quit (Quit: ["Textual IRC Client: www.textualapp.com"])
[5:01] * slang (~slang@173-163-208-195-westflorida.hfc.comcastbusiness.net) Quit (Quit: slang)
[5:26] * adjohn (~adjohn@108-225-130-229.lightspeed.sntcca.sbcglobal.net) Quit (Quit: adjohn)
[5:41] * nhmlap (~nhm@ Quit (Ping timeout: 480 seconds)
[5:45] * buck (~buck@c-24-7-14-51.hsd1.ca.comcast.net) has joined #ceph
[5:46] * Cube2 (~Cube@ Quit (Quit: Leaving.)
[5:53] * sagelap (~sage@ has joined #ceph
[5:56] * miroslav (~miroslav@c-98-248-210-170.hsd1.ca.comcast.net) has joined #ceph
[5:57] * adjohn (~adjohn@108-225-130-229.lightspeed.sntcca.sbcglobal.net) has joined #ceph
[6:01] * ryann (~chatzilla@ has joined #ceph
[6:03] <ryann> I'm sure this is listed in many places (mailing list, etc). I wish to increase the performance of my journals. Do they also need to be on a "journaled" file system. "Do my journals need to be journaled?" :P Can i place them on an ext4?
[6:03] <ryann> ...right now they live ont he same btrfs partition as my osds.
[6:04] <kavonr> journaled filesystem protects you (somewhat) in the event of system failure during writes, I think.
[6:05] <kavonr> I read that you can put your journals on a separate device, particularly ssd, to boost performance. (right now, journal writes may compete with fs writes... may not be much of a problem but it's a risk)
[6:06] <kavonr> (new ceph user, but that's what I've found in the filesystem world... until someone else wakes up to speak more authoritatively :) )
[6:07] * nhmlap (~nhm@253-231-179-208.static.tierzero.net) has joined #ceph
[6:07] <ryann> Sure. I'm researching now, however, #kavonr, or others, do we know if 0.48.1 is able to use complete block dev's? or is that only stable after 0.53/0.54? Perhaps that's the direction i should head. I've been afraid to use anything past 0.48.1 in production.
[6:08] <ryann> ...for journaling. that question was "use complete block devices for journals"
[6:08] <ryann> sorry
[6:08] <kavonr> joshd told me earlier that there's a regression in 0.53 regarding whole block devices, and an easy workaround.
[6:08] <kavonr> "you need to set 'osd journal size = 0' or it won't use the whole block device"
[6:09] <ryann> Ooh! Thanks for that! copy/paste.....
[6:09] <kavonr> I just went from 0.48 to 0.53-1 this evening...
[6:09] <kavonr> he did say that's fixed but 0.53 is latest dev release.
[6:10] <kavonr> now I'm updating ubuntu on my lab hadoop cluster, cos I'm a glutton for punishment and my fiancee is very forgiving.
[6:16] * miroslav (~miroslav@c-98-248-210-170.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[6:16] <buck> anyone about that I could bounce a teuthology question off of?
[6:16] <buck> well, teuthology / gitbuilder
[6:17] * rweeks (~rweeks@ has joined #ceph
[6:27] <kavonr> sorry, not me.
[6:34] <dmick> buck: you can try
[6:34] <buck> I was wondering if there was a "best practice" as far as getting gitbuilder to produce a build with specific configure flags set?
[6:35] <dmick> yeah, I don't know that we have any interface for that exactly
[6:35] <dmick> I mean you can always have teuthology install your own builds (kernel or user)
[6:36] <dmick> and if you can set things up in the Makefiles so that the 'standard' invocation does what you want, I guess you can have the branch build as you like automagically
[6:36] <dmick> (standard invocation of autotools)
[6:38] <buck> As far as pointing teuthology at my own build, is there an example of that sitting around anywhere handy? I think that would solve a few hiccups I've had.
[6:40] <elder> dmick, I think I'll restart my computer in the morning. If you get in a few minutes in advance of our standup maybe we can try getting some form of audio/video going.
[6:40] * adjohn (~adjohn@108-225-130-229.lightspeed.sntcca.sbcglobal.net) Quit (Quit: adjohn)
[6:41] * fzylogic (~fzylogic@ Quit (Quit: fzylogic)
[6:44] * davidz (~Adium@ip68-96-75-123.oc.oc.cox.net) Quit (Quit: Leaving.)
[6:44] * pixel (~pixel@ has joined #ceph
[6:44] * fzylogic (~fzylogic@ has joined #ceph
[6:46] * ryann (~chatzilla@ has left #ceph
[6:48] <dmick> buck: userland: see the ceph.py task documentation
[6:48] <dmick> basically it's path: /path/to/your/build
[6:48] <dmick> elder: ok
[6:48] <buck> dmick: rad, thanks
[6:49] <dmick> it will try to build it, but if you've built it beforehand it'll just be a verify
[6:54] <buck> dmick: that is working like a charm. Thanks a bunch
[6:57] * adjohn (~adjohn@108-225-130-229.lightspeed.sntcca.sbcglobal.net) has joined #ceph
[6:57] <dmick> w00t
[6:58] * adjohn (~adjohn@108-225-130-229.lightspeed.sntcca.sbcglobal.net) Quit ()
[6:58] <pixel> Good morning!
[7:00] <rweeks> good evening!
[7:07] * miroslav (~miroslav@c-98-248-210-170.hsd1.ca.comcast.net) has joined #ceph
[7:09] * miroslav (~miroslav@c-98-248-210-170.hsd1.ca.comcast.net) Quit ()
[7:09] <Guest3170> hi all
[7:09] * Guest3170 is now known as Q310
[7:10] * sjustlaptop (~sam@68-119-138-53.dhcp.ahvl.nc.charter.com) has joined #ceph
[7:15] * buck (~buck@c-24-7-14-51.hsd1.ca.comcast.net) has left #ceph
[7:16] * leaper (40c6ba0a@ircip2.mibbit.com) has joined #ceph
[7:18] * leaper (40c6ba0a@ircip2.mibbit.com) has left #ceph
[7:28] * sjustlaptop (~sam@68-119-138-53.dhcp.ahvl.nc.charter.com) Quit (Ping timeout: 480 seconds)
[7:28] <dmick> hi Q2
[7:28] <dmick> Q310 I mean
[7:33] * rweeks (~rweeks@ Quit (Quit: ["Textual IRC Client: www.textualapp.com"])
[7:42] * nhmlap (~nhm@253-231-179-208.static.tierzero.net) Quit (Ping timeout: 480 seconds)
[7:42] * maxim (~pfliu@ has joined #ceph
[7:46] * Kioob (~kioob@luuna.daevel.fr) has joined #ceph
[7:46] * Kioob (~kioob@luuna.daevel.fr) Quit ()
[7:49] * sagelap (~sage@ Quit (Ping timeout: 480 seconds)
[8:17] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[8:26] * fzylogic (~fzylogic@ Quit (Quit: fzylogic)
[8:39] * prometheanfire (~promethea@rrcs-24-173-105-83.sw.biz.rr.com) has joined #ceph
[8:39] <prometheanfire> so... I should use tmalloc?
[8:41] <prometheanfire> tcmalloc that is
[9:08] * maxim (~pfliu@ Quit (Ping timeout: 480 seconds)
[9:16] * ivoks (~ivoks@jupiter.init.hr) has joined #ceph
[9:19] * dmick (~dmick@ Quit (Ping timeout: 480 seconds)
[9:19] <ivoks> hi; a quick question - is there a reason why one wouldn't consider using corosync's (lib)quorum instead of ceph's? i know that code is not there; i'm just wondering if there is a real blocker to that idea?
[9:20] * dmick (~dmick@2607:f298:a:607:242c:baf9:3d33:d7d8) has joined #ceph
[9:22] * Leseb (~Leseb@ has joined #ceph
[9:25] * ivoks (~ivoks@jupiter.init.hr) Quit (Quit: leaving)
[9:26] * sprog (d4d3c928@ircip4.mibbit.com) has joined #ceph
[9:26] * maxim (~pfliu@ has joined #ceph
[9:26] * verwilst (~verwilst@dD5769628.access.telenet.be) has joined #ceph
[9:26] <sprog> Hi,
[9:33] <sprog> Hey. I'm missing some Information on the ceph Website to understand the whole ceph project. Can someone here please explain how all those ceph components work together. Is there a master node in the cluster? What happens if I only have one mds daemon and that node on which the daemon is running crashes.. and so on..
[9:34] * ivoks (~ivoks@jupiter.init.hr) has joined #ceph
[9:35] * sprog (d4d3c928@ircip4.mibbit.com) Quit (Quit: http://www.mibbit.com ajax IRC Client)
[9:36] * iltisanni (d4d3c928@ircip1.mibbit.com) has joined #ceph
[9:46] * iltisanni (d4d3c928@ircip1.mibbit.com) has left #ceph
[9:46] * iltisanni (d4d3c928@ircip1.mibbit.com) has joined #ceph
[10:03] * jakku (~jakku@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[10:04] * deepsa_ (~deepsa@ has joined #ceph
[10:06] * loicd (~loic@ has joined #ceph
[10:07] * deepsa (~deepsa@ Quit (Ping timeout: 480 seconds)
[10:07] * deepsa_ is now known as deepsa
[10:23] * MikeMcClurg (~mike@cpc10-cmbg15-2-0-cust205.5-4.cable.virginmedia.com) Quit (Ping timeout: 480 seconds)
[10:32] * jakku (~jakku@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[10:32] * jakku (~jakku@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[10:34] * jakku_ (~jakku@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[10:34] * jakku (~jakku@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Read error: Connection reset by peer)
[10:43] <pixel> Does anybody know how to check why the commands `time find . | wc -l` works so slowly??? 83931
[10:43] <pixel> real 10m26.030s
[10:43] <pixel> user 0m0.192s
[10:43] <pixel> sys 0m0.844s
[10:43] * loicd1 (~loic@ has joined #ceph
[10:45] * jakku_ (~jakku@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[10:45] * jakku (~jakku@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[10:49] * loicd (~loic@ Quit (Ping timeout: 480 seconds)
[10:50] <Robe> is lennert den teuling in here?
[10:52] <joao> morning #ceph
[10:53] * tziOm (~bjornar@ has joined #ceph
[10:53] * jakku (~jakku@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Ping timeout: 480 seconds)
[10:54] * BManojlovic (~steki@ has joined #ceph
[10:54] <pixel> hi joao
[10:57] * maxim (~pfliu@ Quit (Quit: Ex-Chat)
[10:58] * LarsFronius (~LarsFroni@testing78.jimdo-server.com) has joined #ceph
[11:03] * match (~mrichar1@pcw3047.see.ed.ac.uk) has joined #ceph
[11:05] * MikeMcClurg (~mike@ has joined #ceph
[11:05] <todin> joao: morning
[11:16] * phil` (~user@ has joined #ceph
[11:20] * MikeMcClurg (~mike@ Quit (Read error: Connection reset by peer)
[11:20] * MikeMcClurg (~mike@firewall.ctxuk.citrix.com) has joined #ceph
[11:22] * yoshi (~yoshi@p37219-ipngn1701marunouchi.tokyo.ocn.ne.jp) Quit (Remote host closed the connection)
[11:50] * MikeMcClurg (~mike@firewall.ctxuk.citrix.com) Quit (Quit: Leaving.)
[11:50] * tryggvil (~tryggvil@163-60-19-178.xdsl.simafelagid.is) Quit (Quit: tryggvil)
[11:50] * MikeMcClurg (~mike@firewall.ctxuk.citrix.com) has joined #ceph
[11:51] * madkiss (~madkiss@ has joined #ceph
[11:56] * jakku (~jakku@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[11:58] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) has joined #ceph
[12:04] * jakku (~jakku@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Ping timeout: 480 seconds)
[12:09] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[12:11] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[12:13] * jakku (~jakku@ac232088.dynamic.ppp.asahi-net.or.jp) has joined #ceph
[12:22] * loicd1 (~loic@ Quit (Ping timeout: 480 seconds)
[12:31] * loicd (~loic@207.209-31-46.rdns.acropolistelecom.net) has joined #ceph
[12:41] * MikeMcClurg1 (~mike@firewall.ctxuk.citrix.com) has joined #ceph
[12:41] * MikeMcClurg (~mike@firewall.ctxuk.citrix.com) Quit (Read error: Connection reset by peer)
[12:45] * MikeMcClurg (~mike@ has joined #ceph
[12:45] * MikeMcClurg1 (~mike@firewall.ctxuk.citrix.com) Quit (Read error: Connection reset by peer)
[12:54] * LarsFronius_ (~LarsFroni@testing78.jimdo-server.com) has joined #ceph
[12:54] * LarsFronius (~LarsFroni@testing78.jimdo-server.com) Quit (Read error: Connection reset by peer)
[12:54] * LarsFronius_ is now known as LarsFronius
[12:54] * LarsFronius (~LarsFroni@testing78.jimdo-server.com) Quit ()
[12:54] <iltisanni> hey there. can anybody here please explain me how all those ceph components work together? Is there a Ceph Master or something like that? And how are OSDs, Pools and PGs working with each other? I don't get it...and can't find something thats helps me understanding on the website
[12:57] * LarsFronius (~LarsFroni@testing78.jimdo-server.com) has joined #ceph
[13:07] <joao> iltisanni, in a nutshell, on the 'backend', you'll have 2/3 major components: monitors, osds and mds's (which are only required for cephfs). osds keep the objects, across pgs, and the pgs are replicated across the osds (if you have replication >= 2)
[13:08] <joao> each pgs has a primary, which is the one the client will communicate with; different pg's may have their primaries in different osds
[13:09] <joao> the client will know which osd to contact given some crush magic and beforehand knowledge of the osdmap, which is obtained from the monitor cluster (thus the client must have knowledge of at least one monitor)
[13:09] <joao> clients contact the monitors for maps, and contact directly the osds for operations
[13:10] <joao> monitors keep track of map changes throughout the cluster, by communicating with osds
[13:10] <joao> monitors do have a master/leader, but that's to guarantee map consistency; if the leader fails, other monitor will take over
[13:12] <joao> same goes for the pg's primaries: if the osd that holds a pg primary replica fails, that osd should be marked down in ~30s, maps will be readjusted, and another osd will take over as that pg's primary (as long as there are available replicas on other osds)
[13:12] <joao> iltisanni, you can also find more infos on this on the articles on the site
[13:12] <pixel> joao: for performance reason should mons, osds and mds be located on different nodes?
[13:13] <joao> pixel, I'd say that you should scatter your daemons across nodes at least for reliability/availability
[13:13] <iltisanni> Thx... Im gonna cut this text out and read it carefully to understand everything :-)
[13:14] <joao> but afaik there is no reason why you shouldn't run monitors and osds on the same nodes; not sure about mds
[13:14] <joao> pixel, I'm not the guy you should talk to about that though
[13:15] <joao> there's people with deployed (test) clusters in the channel, maybe someone can provide some more accurate insight?
[13:15] <iltisanni> what about the mds daemon.. in all sample ceph.conf files only one server acts as mds. What if this Server is going down ?
[13:15] <pixel> I'm newbee in ceph, that is why we need to get all usefull info
[13:16] <joao> pixel, my experience with 'deploying' ceph is usually for dev purposes and in at most a couple of machines
[13:16] <joao> so I've no real idea on the impact real workloads have on the daemons
[13:17] <joao> and don't really want to give you misleading informations :)
[13:17] <pixel> it's ok I'am in testing phase now
[13:18] <joao> iltisanni, you can have several mds; I *think* the mds's will have one 'master' and the other will be on standby in case it fails, so one will take over
[13:18] <iltisanni> ok.. and how can i determine the master?
[13:18] <joao> but I'm not really sure about that as I have not taken a deep look into the mds
[13:18] <joao> yeah... no idea :)
[13:19] <iltisanni> :-) OK
[13:20] <Fruit> iltisanni: the mds that uses the most cpu, in my experience :)
[13:20] <joao> lol
[13:20] <Fruit> or rather the one that is pegged at 100% cpu :\
[13:23] <joao> does anyone know if it is wise to buy an umbrella as soon as one sets foot in amsterdam?
[13:26] * phil` is now known as phil_
[13:27] * LarsFronius_ (~LarsFroni@testing78.jimdo-server.com) has joined #ceph
[13:33] * LarsFronius (~LarsFroni@testing78.jimdo-server.com) Quit (Ping timeout: 480 seconds)
[13:33] * LarsFronius_ is now known as LarsFronius
[13:35] * tryggvil_ (~tryggvil@rtr1.tolvusky.sip.is) has joined #ceph
[13:42] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) Quit (Ping timeout: 480 seconds)
[13:43] <iltisanni> so there is a single point of failure when I only have one mds daemon on one node...?
[13:43] * tryggvil_ (~tryggvil@rtr1.tolvusky.sip.is) Quit (Ping timeout: 480 seconds)
[13:45] <Fruit> iltisanni: yes. so don't do that :)
[13:46] <iltisanni> ok thx
[13:54] * irctc018 (~3ec4141c@2600:3c00::2:2424) has joined #ceph
[13:54] * irctc018 (~3ec4141c@2600:3c00::2:2424) Quit ()
[13:54] * irctc228 (~3ec4141c@2600:3c00::2:2424) has joined #ceph
[14:00] * Moro (~Moro@ has joined #ceph
[14:01] * irctc228 (~3ec4141c@2600:3c00::2:2424) Quit (Quit: TheGrebs.com CGI:IRC)
[14:04] * LarsFronius (~LarsFroni@testing78.jimdo-server.com) Quit (Remote host closed the connection)
[14:04] * LarsFronius (~LarsFroni@testing78.jimdo-server.com) has joined #ceph
[14:16] * BManojlovic (~steki@ Quit (Quit: Ja odoh a vi sta 'ocete...)
[14:18] * madkiss (~madkiss@ Quit (Quit: Leaving.)
[14:20] * Moro (~Moro@ has left #ceph
[14:21] * MoroIta (~MoroIta@ has joined #ceph
[14:24] * LarsFronius_ (~LarsFroni@testing78.jimdo-server.com) has joined #ceph
[14:24] * LarsFronius (~LarsFroni@testing78.jimdo-server.com) Quit (Read error: Connection reset by peer)
[14:24] * LarsFronius_ is now known as LarsFronius
[14:42] * MoroIta (~MoroIta@ Quit (Ping timeout: 480 seconds)
[14:57] * loicd1 (~loic@207.209-31-46.rdns.acropolistelecom.net) has joined #ceph
[14:57] * loicd (~loic@207.209-31-46.rdns.acropolistelecom.net) Quit (Read error: Connection reset by peer)
[14:58] * andret (~andre@pcandre.nine.ch) Quit (Ping timeout: 480 seconds)
[15:00] * aliguori (~anthony@cpe-70-123-145-75.austin.res.rr.com) has joined #ceph
[15:08] * loicd1 (~loic@207.209-31-46.rdns.acropolistelecom.net) Quit (Ping timeout: 480 seconds)
[15:10] * nhorman (~nhorman@hmsreliant.think-freely.org) has joined #ceph
[15:22] * scalability-junk (~stp@188-193-208-44-dynip.superkabel.de) has joined #ceph
[15:42] * dspano (~dspano@rrcs-24-103-221-202.nys.biz.rr.com) has joined #ceph
[15:43] * tryggvil (~tryggvil@vmc-213-220-78-180.3G.internet.is) has joined #ceph
[15:43] * tryggvil (~tryggvil@vmc-213-220-78-180.3G.internet.is) Quit ()
[15:44] * loicd (~loic@207.209-31-46.rdns.acropolistelecom.net) has joined #ceph
[15:46] * nhmlap (~nhm@253-231-179-208.static.tierzero.net) has joined #ceph
[15:53] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) has joined #ceph
[15:54] * miroslav (~miroslav@173-228-38-131.dsl.dynamic.sonic.net) has joined #ceph
[15:58] * pixel (~pixel@ Quit (Remote host closed the connection)
[15:59] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) Quit (Remote host closed the connection)
[16:00] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) has joined #ceph
[16:01] <prometheanfire> should tcmalloc be used when possible in ceph?
[16:03] * madkiss (~madkiss@ has joined #ceph
[16:03] <wido> prometheanfire: I'd advise to do so. It will save you a large amount of memory usage
[16:04] * rlr219 (43c87e04@ircip4.mibbit.com) Quit (Quit: http://www.mibbit.com ajax IRC Client)
[16:04] * rlr219 (43c87e04@ircip4.mibbit.com) has joined #ceph
[16:06] * PerlStalker (~PerlStalk@perlstalker-1-pt.tunnel.tserv8.dal1.ipv6.he.net) has joined #ceph
[16:07] * tziOm (~bjornar@ Quit (Remote host closed the connection)
[16:09] * Qu310 (Q@qten.qnet.net.au) Quit ()
[16:10] * lofejndif (~lsqavnbok@82VAAHG8I.tor-irc.dnsbl.oftc.net) has joined #ceph
[16:16] <prometheanfire> ok, I did compile it with it, just curious :D
[16:29] <jamespage> .query scuttlemonkey_
[16:30] * prometheanfire (~promethea@rrcs-24-173-105-83.sw.biz.rr.com) has left #ceph
[16:40] * noob2 (a5a00214@ircip1.mibbit.com) has joined #ceph
[16:45] * MikeMcClurg (~mike@ Quit (Ping timeout: 480 seconds)
[16:47] * scuttlemonkey_ (~scuttlemo@c-69-244-181-5.hsd1.mi.comcast.net) Quit (Quit: This computer has gone to sleep)
[16:49] * vata (~vata@ has joined #ceph
[16:51] <noob2> can you create pool names with _ in them?
[17:00] * verwilst (~verwilst@dD5769628.access.telenet.be) Quit (Quit: Ex-Chat)
[17:15] * MikeMcClurg (~mike@ has joined #ceph
[17:16] <rlr219> mikeryan: you in??
[17:49] * anna1 (~Adium@ has joined #ceph
[17:52] * nwatkins1 (~Adium@soenat3.cse.ucsc.edu) has joined #ceph
[17:52] * loicd (~loic@207.209-31-46.rdns.acropolistelecom.net) Quit (Ping timeout: 480 seconds)
[17:52] * lofejndif (~lsqavnbok@82VAAHG8I.tor-irc.dnsbl.oftc.net) Quit (Quit: gone)
[17:54] * madkiss (~madkiss@ Quit (Quit: Leaving.)
[18:00] * fzylogic (~fzylogic@ has joined #ceph
[18:00] * fzylogic (~fzylogic@ Quit ()
[18:01] * davidz (~Adium@ip68-96-75-123.oc.oc.cox.net) has joined #ceph
[18:07] <noob2> so i had a 1 osd cluster of ceph and i added a second osd. It seems like it started to scrub everything and then stopped. I'm not sure why. I see this as my last message: 2012-10-26 11:38:16.321406 mon.0 [INF] pgmap v341: 192 pgs: 168 active+clean, 9 active+remapped, 15 active+degraded; 2134 MB data, 6179 MB used, 96170 MB / 102349 MB avail; 66/1084 degrad
[18:11] * sagelap (~sage@216.sub-70-197-147.myvzw.com) has joined #ceph
[18:17] <noob2> i instructed ceph to scrub again and told it to repair
[18:17] <noob2> doesn't seem to want to repair that last 6%
[18:23] * match (~mrichar1@pcw3047.see.ed.ac.uk) Quit (Quit: Leaving.)
[18:28] * LarsFronius (~LarsFroni@testing78.jimdo-server.com) Quit (Ping timeout: 480 seconds)
[18:29] * loicd (~loic@magenta.dachary.org) has joined #ceph
[18:38] * nhmlap (~nhm@253-231-179-208.static.tierzero.net) Quit (Ping timeout: 480 seconds)
[18:40] <rlr219> mikeryan, sjust: either of you available?
[18:41] <sjust> rlr219: here
[18:42] <rlr219> sjust; hi, I dont know if you or mike were going to help me try to repair my pg on my cluster. just checking cause I am going out of town later.
[18:42] * Leseb (~Leseb@ Quit (Quit: Leseb)
[18:42] <sjust> I think mike had a plan
[18:43] <sjust> how much longer will you be around?
[18:43] * sagelap (~sage@216.sub-70-197-147.myvzw.com) Quit (Ping timeout: 480 seconds)
[18:43] <rlr219> until about 3 CST
[18:43] <sjust> k, mike should be in shortly
[18:44] <rlr219> ok. I will be patient then. Thanks! :-)
[18:44] <sjust> I think he needed the list of clone sizes?
[18:44] * sagelap (~sage@2607:f298:a:607:aca0:7479:696a:6818) has joined #ceph
[18:44] * Solver (~robert@atlas.opentrend.net) Quit (Remote host closed the connection)
[18:44] * Solver (~robert@atlas.opentrend.net) has joined #ceph
[18:44] <rlr219> I sent him that last night after I got home. I got them from all 3 OSDs.
[18:45] <sjust> ok
[19:00] * madkiss (~madkiss@ has joined #ceph
[19:00] * sagewk (~sage@ Quit (Read error: Connection reset by peer)
[19:01] <noob2> with ceph do all servers in the cluster need a mon and a ceph directory created before running mkcephfs?
[19:01] <noob2> i thought only the osd's needed a osd and only the mon's needed the mon directory
[19:01] <dspano> I just read about injectargs. That's awesome!
[19:03] <PerlStalker> It is possible to say that one pool is on one set of hosts and another is on another set?
[19:09] * chutzpah (~chutz@ has joined #ceph
[19:09] <elder> joshd, is there a way currently on the client to get an rbd image *name* given an image *id*? (Haven't looked, just looking for a quick answer)
[19:13] * sjustlaptop (~sam@2607:f298:a:607:804c:5162:8abe:b99) has joined #ceph
[19:13] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[19:15] * sagewk (~sage@2607:f298:a:607:bcf7:887f:817:c0d7) has joined #ceph
[19:16] <elder> It looks like only format 2 image snapshots can serve as parents for clones.
[19:16] <elder> I thought maybe the top level image could have been format one.
[19:17] * nhmlap (~nhm@ has joined #ceph
[19:20] * scuttlemonkey (~scuttlemo@ has joined #ceph
[19:25] * madkiss (~madkiss@ Quit (Quit: Leaving.)
[19:26] <joshd> elder: yes, the rbd_directory has a method to get a name given an id, and vice versa
[19:26] <joshd> elder: only format 2 snapshots since only they can be protected
[19:31] * sjustlaptop (~sam@2607:f298:a:607:804c:5162:8abe:b99) Quit (Ping timeout: 480 seconds)
[19:32] * scuttlemonkey_ (~scuttlemo@adsl-76-226-58-145.dsl.sfldmi.sbcglobal.net) has joined #ceph
[19:38] * scuttlemonkey (~scuttlemo@ Quit (Ping timeout: 480 seconds)
[19:43] <nwatkins1> Is there an option in gui gui to force merge commit like —no-ff
[19:45] <sagewk> nwatkins1: dunno, never used git gui for merge/pull.. only for commit and amend
[19:46] <nwatkins1> sagewk: I guess I can't have everything. amend will work, though. thx
[19:47] <sagewk> nwatkins1: oh, you're running multiple mds's.. try with 1 and see if you see the same behavior.
[19:47] <nwatkins1> sagewk: that experiment should have been with 1
[19:47] <sjust> rlr219: it is looking like you have two corrupted btrfs volumes
[19:48] <sjust> we think we have a bandaid to get them back on their feet long enough to pull the data off to other osds
[19:48] * MikeMcClurg (~mike@ Quit (Ping timeout: 480 seconds)
[19:48] * loicd (~loic@ has joined #ceph
[19:49] <sagewk> nwatkins1: not the logs you sent.. the client is talking to two different mds's, about an hour before the mds log starts
[19:50] * miroslav (~miroslav@173-228-38-131.dsl.dynamic.sonic.net) Quit (Quit: Leaving.)
[19:50] * adjohn (~adjohn@ has joined #ceph
[19:50] <nwatkins1> sagewk: I think something is wonky. I nuked all the logs before running the test. Lemme see if I grabbed the wrong stuff..
[19:53] <joao> f
[19:53] <joao> oops
[19:56] <elder> joshd, ok, thanks. I"m not sure I need it. dir_get_name, right?
[19:56] <elder> Not a big deal.
[19:56] <elder> Also only format 2 snapshots because format 1 has no id.
[19:56] <elder> ...which is required to specify the parent.
[19:58] * loicd (~loic@ Quit (Read error: No route to host)
[19:58] <nwatkins1> sagewk: try those logs again—should be fixed with one mds
[19:58] * loicd (~loic@ has joined #ceph
[19:59] <joshd> elder: yes, dir_get_name
[19:59] <dmick> noob2: apparently _ in pool names is fine.
[19:59] * nwatkins1 is now known as noah
[20:00] * noah is now known as noahdesu
[20:00] <dmick> and mkcephfs *ought* to be creating directories
[20:00] <dmick> although it's worth pointing out that ceph-deploy is the newer "standalone" tool
[20:01] * elder (~elder@c-71-195-31-37.hsd1.mn.comcast.net) Quit (Quit: Leaving)
[20:04] <gregaf> PerlStalker: yes; you'd want a custom CRUSH map that puts the OSDs in different "buckets" and point the pools to the correct buckets
[20:05] <gregaf> noob2: you're correct, the monitors are the only ones who need a monitor directory; the OSDs are the only ones who need an OSD directory...
[20:05] * elder (~elder@c-71-195-31-37.hsd1.mn.comcast.net) has joined #ceph
[20:05] <sagewk> joshd: see wip-rbd-completionrace
[20:05] <gregaf> the thing with your degraded PGs is probably because you've only got two nodes and the default CRUSH rules don't cope with that very well
[20:06] <gregaf> (in general, if you have a number of buckets very close to your replication count, CRUSH isn't going to like it because it's a pseudo-random distribution)
[20:07] <dmick> sagewk: ah, that was that Eureka moment
[20:07] <dmick> cool
[20:07] * dmick (~dmick@2607:f298:a:607:242c:baf9:3d33:d7d8) Quit (Remote host closed the connection)
[20:10] * dmick (~dmick@2607:f298:a:607:dd60:ae90:5409:88f1) has joined #ceph
[20:10] <joshd> sagewk: that makes sense, it probably fixes #3413 too
[20:10] <sagewk> nwatkins1: this log has a ~4 second mkdirs.. am i looking at the right thing?
[20:11] * scuttlemonkey_ is now known as scuttlemonkey
[20:11] <sagewk> joshd,dmick: ooh.. wanna see if it does dmick?
[20:14] <noahdesu> sagewk: in client.log i'm seeing from line 862 to 1222 a mkdirs lasting from 10:53:17 to 10:54:21
[20:14] <noob2> has anyone else had problems with ceph vm clients rebooting and corrupting the rbd pool?
[20:17] <benpol> I see warnings about using tmpfs for osd journals (ie "just for testing"), what sort of problems (aside from the obvious need to recreate journals after a reboot) are there with journals in tmpfs?
[20:21] <noob2> does anyone know how you get ceph out of a degraded state?
[20:21] * loicd (~loic@ Quit (Ping timeout: 480 seconds)
[20:25] <sjust> noob2: what is the output of ceph -s?
[20:26] <noob2> http://mibpaste.com/3BmO5q
[20:26] <noob2> we fired up 10 vm's to do bonnie++ testing against my 2 osd's and 1 monitor
[20:26] <noob2> the 10 vm's crashed and we hard rebooted them
[20:26] <sjust> noob2: they are degraded because you lost an osd
[20:26] <sjust> one of the osds is dead
[20:26] <noob2> oh :(
[20:26] <sjust> osdmap e7: 2 osds: 1 up, 1 in
[20:26] <noob2> oh...
[20:26] <sjust> so, 1 osd is down
[20:26] <noob2> sorry
[20:26] <noob2> didn't see that
[20:27] <sjust> ceph osd dump will tell you which one is dead
[20:27] <sjust> no problem
[20:27] <noob2> so after a restart will it scrub itself?
[20:27] <noob2> restart of the ceph process i mean
[20:28] <sjust> it'll recover the osd, if that's what you mean
[20:28] <noob2> yeah
[20:28] <sjust> it'll scrub as well, but that's a different thing
[20:28] <noob2> i'm watching ceph -w
[20:28] <sjust> is the osd back up?
[20:28] <noob2> yup
[20:28] <sjust> cool
[20:28] <noob2> i'm still wondering how we killed it
[20:28] <noob2> i have 2 vm's with fibre backed drives playing osd
[20:28] <noob2> and 10 vm's trying to bonnie++ test it
[20:29] <noob2> i have my journal set to 1000 and just 10 3GB disks
[20:29] <noob2> 10 3GB rbd's i mean
[20:29] <sagewk> joshd: you're right, this only affects master (post-striping)
[20:30] <joshd> that's a relief
[20:30] <dmick> btw sagewk, tracker was unresponsive last night
[20:30] <dmick> I killed both ruby processes and it came back
[20:32] * MikeMcClurg (~mike@cpc10-cmbg15-2-0-cust205.5-4.cable.virginmedia.com) has joined #ceph
[20:35] <noob2> sjust: 8 out of 10 vms' now can't map their rbd disks anymore
[20:35] <noob2> is there anyway to repair them?
[20:35] <sjust> what is the output of ceph -s?
[20:35] <noob2> health OK and 2 up, 2 in
[20:36] <sjust> I need the pg line
[20:36] <sjust> ^pgmap.*
[20:36] <noob2> pgmap v365: 384 pgs: 384 active+clean; 2724 MB data, 7451 MB used, 94898 MB / 102349 MB avail
[20:36] <sjust> ok
[20:36] <sjust> and they can't map the rbd volumes?
[20:36] <sjust> joshd: any thoughts?
[20:36] <noob2> when he tries to map them the command just hangs
[20:36] <sagewk> noahdesu: oh i see
[20:36] <noob2> he being my coworker
[20:37] <dmick> a little confused by "vms can't map their rbd images"
[20:37] <noob2> ok forget that they're vm's
[20:37] <noob2> think of them as a regular unix box
[20:37] <dmick> is that ceph -s running on the vm?
[20:37] <noob2> and i'm using the rbd kernel module to map the drives
[20:37] <dmick> no, I get it
[20:37] * scuttlemonkey (~scuttlemo@adsl-76-226-58-145.dsl.sfldmi.sbcglobal.net) Quit (Quit: This computer has gone to sleep)
[20:37] <dmick> that wasn't the confusion
[20:37] <noob2> oh ok
[20:37] <noob2> sorry
[20:38] <noob2> i'm just pretending these are regular physical servers and rbd is their drive
[20:38] <noahdesu> sagewk: yeh, there are a bunch of mkdirs in there from the unit tests, but its the first one in the log.
[20:38] <dmick> noob2: get it. Is the ceph -s running successfully on the vm's that can't map?
[20:38] <noob2> let me check
[20:39] <sagewk> noahdesu: oh
[20:39] <sagewk> noahdesu: there was another client just recently connected that didn't unmount cleanly, so it's waiting for their session to time out and go stale.
[20:40] <sagewk> if you want 60 seconds before running, or otherwise ensure that there wasn't a client that just crashed, and then run your test, it shouldn't hang like that
[20:40] <sagewk> this is the price you pay for stateful sessions, locks/leases, and strong consistency :)
[20:40] * dweazle (~dweazle@tilaa.krul.nu) has joined #ceph
[20:41] <sagewk> (in the mds log, there is a -- --> -- client_caps(revoke ino 1 1 seq 2 caps=pAsLsXs ... message sent, and that client is apparnetly gone
[20:41] <noob2> dmick: 2012-10-26 14:41:06.620221 7f418f873700 monclient: hunting for new mon
[20:41] <noob2> they're just spitting out that
[20:41] <dmick> ah, ok, so they've lost contact with the cluster, is the problem
[20:42] <dmick> network connectivity with the monitor(s) in the ceph.conf?
[20:42] <sjust> noob2: what is the ^monmap.* line/
[20:42] <sjust> ?
[20:42] <sjust> in ceph -s
[20:42] <noob2> checking
[20:43] <noahdesu> sagewk: is the granularity at the FS level like file,dir .. or something higher level?
[20:45] * Cube (~Cube@ has joined #ceph
[20:46] <noob2> sjust: grep doesn't return anything
[20:46] <noob2> i just get hunting for new mon
[20:47] <noob2> i rebooted them and got the same
[20:47] <sjust> where did you run ceph -s before?
[20:47] <noob2> i rmmod rbd, then modprobe and it does the same
[20:47] <noob2> i ran it on the same servers
[20:47] <noob2> worked ok before
[20:48] <dmick> noob2: can you ping the monitors from the vms?
[20:48] <sjust> the problem is that the ceph client cannot talk to the monitors
[20:48] <noob2> yeah let me try
[20:48] <noob2> i see this in the osd that died: [ 3932.116328] Out of memory: Kill process 17556 (ceph-osd) score 883 or sacrifice child [ 3932.116402] Killed process 17556 (ceph-osd) total-vm:1332300kB, anon-rss:430124kB, file-rss:0kB
[20:48] <sjust> ok, it got oom'd
[20:48] <noob2> looks like i ran out of ram
[20:48] <dweazle> sjust: djees, how many irc client have you got running :)
[20:48] <dweazle> +s
[20:49] <sjust> more than one
[20:50] <noob2> sjust: yeah the vm's can ping the monitor just fine
[20:50] <noob2> monitor says its running
[20:51] <noob2> maybe i should shutdown this cluster and bump up the ram before continuing
[20:52] * justinwarner (~Thunderbi@ Quit (Remote host closed the connection)
[20:56] * scalability-junk (~stp@188-193-208-44-dynip.superkabel.de) Quit (Ping timeout: 480 seconds)
[20:56] * jmlowe (~Adium@c-71-201-31-207.hsd1.in.comcast.net) has joined #ceph
[20:56] <jmlowe> joshd: Did you do the openstack integration?
[20:56] <joshd> yes
[20:58] <jmlowe> I'm attempting an openstack install, is there an obvious mistake I'm making based on this error:
[20:58] <jmlowe> File "/usr/lib/python2.7/dist-packages/glance/api/v1/images.py", line 437, in _upload
[20:58] <jmlowe> image_meta['size'])
[20:58] <jmlowe> File "/usr/lib/python2.7/dist-packages/glance/store/rbd.py", line 224, in add
[20:58] <jmlowe> with rados.Rados(conffile=self.conf_file, rados_id=self.user) as conn:
[20:58] <jmlowe> File "/usr/lib/python2.7/dist-packages/rados.py", line 127, in __enter__
[20:58] <jmlowe> self.connect()
[20:58] <jmlowe> File "/usr/lib/python2.7/dist-packages/rados.py", line 185, in connect
[20:58] <jmlowe> raise make_ex(ret, "error calling connect")
[20:58] <jmlowe> ObjectNotFound: error calling connect
[21:01] * loicd (~loic@magenta.dachary.org) has joined #ceph
[21:01] <noob2> sjust: a reboot of each osd fixed the problem
[21:03] <joshd> jmlowe: that's kind of a strange error to get from connect (it's ENOENT). does your ceph.conf that glance is pointing at have the monitor addresses?
[21:04] <jmlowe> I think so, ceph commands work
[21:04] <joshd> including as the user that glance is using? assuming you're using cephx
[21:05] <joshd> what's your glance-api.conf have for rbd options?
[21:06] <elder> joshd, if I wanted to look up an image's name given its id, what directory object would I use?
[21:07] * buck (~buck@bender.soe.ucsc.edu) has joined #ceph
[21:07] <joshd> elder: there's one per pool name 'rbd_directory'. there's a RBD_DIRECTORY #define in rbd_types.h
[21:08] <elder> OK.
[21:08] <elder> I've seen that. Makes sense.
[21:10] <jmlowe> joshd: looks like I am having some auth problems with the glance user
[21:10] <jmlowe> auth: failed to decode key '[client.images]
[21:12] <PerlStalker> gregaf: Are there docs that show such a setup?
[21:12] <joshd> jmlowe: it sounds like you specified keyfile when you meant keyring?
[21:13] <joshd> jmlowe: the '[client.$id]' thing is only in keyring files
[21:13] <jmlowe> joshd: yep, operation not permitted, maybe I didn't authorize that key for that pool?
[21:14] <joshd> jmlowe: possibly, check ceph auth list
[21:14] <jmlowe> caps: [mon] allow r
[21:14] <jmlowe> caps: [osd] allow rwx pool=images
[21:15] <jmlowe> su -c 'rbd --keyring /etc/ceph/ceph.client.images.keyring ls /images' glance
[21:15] <jmlowe> 2012-10-26 15:15:07.026495 7f7d5a0a6780 0 librados: client.admin authentication error (1) Operation not permitted
[21:15] <jmlowe> error: couldn't connect to the cluster!
[21:16] <jmlowe> same thing with the correct pool name images
[21:16] <joshd> jmlowe: that's still trying to use client.admin, you need to try 'rbd --keyring /etc/ceph/ceph.client.images.keyring --id images ls images'
[21:17] <jmlowe> su -c 'rbd --id images --keyring /etc/ceph/ceph.client.images.keyring ls images' glance
[21:17] <jmlowe> error: (1) Operation not permitted
[21:17] <jmlowe> slightly better?
[21:18] <joshd> yeah, so you either have the wrong key in that keyring file or the glance user can't read the keyring file
[21:18] <jmlowe> this seems to work su -c 'cat /etc/ceph/ceph.client.images.keyring' glance
[21:18] <jmlowe> so wrong key?
[21:19] <joshd> sounds like it
[21:21] <jmlowe> key matches the one in auth list, I must be missing something
[21:22] <joshd> ok, try adding --debug-ms 1 --debug-auth 20 --debug-monc 1 --log-file /path/to/log
[21:24] <jmlowe> where do I add that?
[21:25] <joshd> i.e. su -c 'rbd --id images --keyring /etc/ceph/ceph.client.images.keyring ls images --debug-ms 1 --debug-auth 20 --debug-monc 1 --log-file /path/to/log' glance
[21:26] <joshd> if you could put the log on a pastebin-type site that'd be great
[21:28] <jmlowe> http://pastebin.com/90nAE5Ed
[21:31] <joshd> ok, so it's connecting fine, but the osd is rejecting the first i/o request with 'operation not permitted'
[21:32] <jmlowe> mon and osd are out of sync in some way?
[21:32] <dmick> maybe all the daemons don't have the same keyring file?...
[21:33] <joshd> no, keyring file has nothing to do with it, and the mons and osds can't get out of sync when the client has the latest map from the mons
[21:33] <joshd> what version are your osds?
[21:34] <jmlowe> ceph version 0.53 (commit:2528b5ee105b16352c91af064af5c0b5a7d45d7c)
[21:36] <joshd> ah, ok
[21:37] <joshd> I'd thought all my osd cap parsing bug fixes made it into 0.53, but they did not
[21:37] <joshd> you'll need to make the osd caps 'allow rwx pool images'
[21:38] <joshd> the '=' is accepted again in the next version
[21:41] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[21:44] * loicd (~loic@magenta.dachary.org) has joined #ceph
[21:44] <buck> Question for the room: I'm working on integrating test cases for the Java bindings. Right now, they exist as .class files in the ceph/src/java directory and are executed via ant. I'm thinking of moving them into a jar file ( libcephfs-test.jar ?) and then teuthology will pull it down with the other binaries (assuming the build used the correct flags to include java support). Thoughts?
[21:45] <dmick> seems like upside with no downside
[21:46] <joshd> buck: sounds good to me
[21:47] <noahdesu> buck: how should we handle the dependency on junit? it's needed when building the jar, and then again later when you want to run it.
[21:47] <joshd> noahdesu: it's not a problem to add more packages to ceph-qa-chef.git
[21:47] <buck> noahdesu: good question. For teuthology, I can wget it but in the general case, a user would need to have installed it some where
[21:48] * slang (~slang@173-163-208-195-westflorida.hfc.comcastbusiness.net) has joined #ceph
[21:48] <buck> joshd: what's protocol for informing users they need that package if they want to run the tests themselves? Make junit required for the ceph-java package?
[21:49] <jmlowe> ok, I give up, how to I set the caps
[21:49] <joshd> buck: generally test dependencies don't get included with normal package requires. if building the tests requires junit, that's good enough
[21:49] * danieagle (~Daniel@ has joined #ceph
[21:49] <buck> joshd: cool. Thanks for the info.
[21:50] <dmick> buck: sounds like the package is required during build? If so, add it to the README
[21:50] <noahdesu> joshd: can those dependencies be setup to go download a jar file from, say, github?
[21:50] <buck> dmick: that's correct (AFAIK) and will do
[21:51] <joshd> noahdesu: tests don't really have formal dependencies unless they're included in some package, and in that case it's up to that packaging system's conventions
[21:51] <dmick> noahdesu: build dependencies are expressed in the README and in the ceph-qa-chef code
[21:51] <buck> er wait.... if we want to build the test .jar with the java .jar files, we'll need junit at that point, so it is a build dependency
[21:52] <buck> so yeah, it goes in the README for source builds and chef for automated builds....right?
[21:52] <dmick> well sorry, runtime dependencies are in the chef
[21:52] <joshd> jmlowe: remove and re-add client.images
[21:52] <noahdesu> joshd: gotcha. i think ubuntu has good standards for java and classpath magic (hopefully..)
[21:52] <dmick> teuthology machines don't build. autobuild-ceph might need some updating
[21:53] * scalability-junk (~stp@188-193-208-44-dynip.superkabel.de) has joined #ceph
[21:53] <dmick> not sure just how gitbuilders get required packages installed
[21:54] <gregaf> PerlStalker: I don't think there are examples, but you should be able to wrangle it out of the docs
[21:54] <gregaf> http://ceph.com/docs/master/cluster-ops/crush-map/#crush-maps
[21:54] <dmick> buck: I guess fabfile.py. see _rpm_install and _apt_install, and callers
[21:55] <jmlowe> Success!
[21:55] <joshd> jmlowe: cool, sorry for the trouble. some but not all of my bug fixes got included in 0.53, which lead to this problem
[21:56] <buck> dmick: ok. I'll check that out
[21:57] <PerlStalker> gregaf: Thanks. I've been reading that page. I'll where that gets me.
[21:58] <dmick> buck: the thing I'm not sure about is that the gitbuilders are only initialized with fabric/fabfile.py. I don't know how you adjust running gitbuilders. Perhaps only manually.
[21:58] <PerlStalker> s/where/see where
[21:58] <dmick> sagewk probably knows
[21:58] * miroslav (~miroslav@173-228-38-131.dsl.dynamic.sonic.net) has joined #ceph
[21:58] <dmick> maybe glowell or glowell1
[21:58] * miroslav (~miroslav@173-228-38-131.dsl.dynamic.sonic.net) Quit (Read error: Connection reset by peer)
[21:58] * miroslav (~miroslav@c-98-248-210-170.hsd1.ca.comcast.net) has joined #ceph
[22:02] * slang (~slang@173-163-208-195-westflorida.hfc.comcastbusiness.net) Quit (Quit: slang)
[22:02] <jmlowe> ok, so I still have the glance-api error of ObjectNotFound: error calling connect
[22:05] * Leseb (~Leseb@5ED17881.cm-7-2b.dynamic.ziggo.nl) has joined #ceph
[22:05] <glowell1> dmick: fabric/fabfile.py just does the initial install. It does set up everything needed for the default builds (there are several flavors). There is a list of needed packages that gets installed using apt-get, yum, or zypper as appropriate for the platform.
[22:07] <dmick> yeah, that's what I thought. So if we have a new dependency, it's manual update?
[22:08] * rweeks (~rweeks@ has joined #ceph
[22:09] <PerlStalker> By using multiple 'step choose' entries in a rule, I can say that I want an object in X racks and then on Y hosts in those racks. Right?
[22:11] <joshd> jmlowe: try add debug ms = 1 debug auth = 20 debug monc = 1 log file = /blah to the [client.images] section of your ceph.conf
[22:12] <glowell1> The profiles in the fabfile.py can be updated and the install re-run. I'd be hesitant to do that with the existing gitbuilders because I don't know for sure that are no manual changes that wold get stepped on.
[22:12] <joshd> PerlStalker: yes, exactly. also you probably want chooseleaf rather than choose
[22:13] <PerlStalker> joshd: What's the difference? I don't see chooseleaf in the docs though it is in my mostly default crushmap.
[22:14] <joshd> PerlStalker: chooseleaf will go all the way down to a leaf node, while choose may not, and may run into problems that the non-default crust tunables solve
[22:14] <PerlStalker> joshd: Ah. chooseleaf it is then.
[22:14] <joshd> s/crust/crush/
[22:16] <PerlStalker> How does the number of replicas interact with the number of buckets set with firstn?
[22:17] * Tamil (~Adium@ has joined #ceph
[22:18] <joshd> the first R buckets emitted are used as replicas
[22:18] <joshd> when replication level R is used
[22:18] <PerlStalker> Makes sense.
[22:21] <PerlStalker> So, if I have a hierarchy created by a ruleset that has 2 racks, each with 10 hosts, and a replication level R=2, then a replica of an object will be on exactly one host in each rack.
[22:23] <jmlowe> joshd: [client.images] section?
[22:24] * Leseb_ (~Leseb@ has joined #ceph
[22:24] <joshd> jmlowe: you can add a [client.images] section if you don't have one. it'll work if it's just in a global [client] section as well.
[22:24] <joshd> jmlowe: it's analogous to [osd] and [osd.1]
[22:25] <joshd> jmlowe: and just to be clear, this is only on the node running glance
[22:26] * Leseb (~Leseb@5ED17881.cm-7-2b.dynamic.ziggo.nl) Quit (Read error: Connection reset by peer)
[22:26] * Leseb (~Leseb@5ED17881.cm-7-2b.dynamic.ziggo.nl) has joined #ceph
[22:28] <jmlowe> hmm, I don't seem to be getting the logfile
[22:29] <joshd> PerlStalker: if you just want to separate replicas across racks, you can have a hierarchy going root -> rack -> host -> device, and say 'step chooseleaf firstn 0 type rack', then 'step emit'
[22:29] * justinwarner (~Thunderbi@ has joined #ceph
[22:31] <jmlowe> nm, there it is
[22:31] <jmlowe> http://pastebin.com/PGZvWJq1
[22:33] * nhorman (~nhorman@hmsreliant.think-freely.org) Quit (Quit: Leaving)
[22:33] <joshd> jmlowe: hmm, no problems there... it looks like the 'debug ms = 1' didn't take effect though? might as well add 'debug rados = 20' for good measure
[22:34] * Leseb_ (~Leseb@ Quit (Ping timeout: 480 seconds)
[22:36] <jmlowe> mean anything? : 2012-10-26 16:35:09.512996 7fc883f49780 10 librados: Objecter returned from read r=-2
[22:37] <joshd> jmlowe: yeah, that's the error code it's reporting... what object is it trying to read (should say in a message above that)
[22:37] <jmlowe> 2012-10-26 16:35:09.509152 7fc883f49780 1 librados: waiting for osdmap
[22:37] <jmlowe> 2012-10-26 16:35:09.510255 7fc883f49780 1 librados: init done
[22:37] <jmlowe> 2012-10-26 16:35:09.511142 7fc87c35e700 10 cephx client: build_authorizer for service osd
[22:39] <joshd> that still doesn't have ms (messenger) debugging for some reason
[22:40] <jmlowe> doh, typo on my part had 'ms =1' instead of 'debug ms = 1'
[22:40] <joshd> ah, it's kind of silly that that matters
[22:42] <jmlowe> *sigh* now I'm not getting a log at all
[22:42] <joshd> maybe restart glance-api to be sure it's re-reading the config file?
[22:43] <jmlowe> did that a few times
[22:44] * BManojlovic (~steki@ has joined #ceph
[22:45] <jmlowe> well, I've exhausted my patience for today
[22:45] <jmlowe> I'll try again monday
[22:46] <joshd> ok, have a good weekend
[22:46] <rlr219> mikeryan: you in?
[22:56] <rlr219> sjust: you there?
[22:56] * lofejndif (~lsqavnbok@83TAAB1A6.tor-irc.dnsbl.oftc.net) has joined #ceph
[23:00] * jmlowe (~Adium@c-71-201-31-207.hsd1.in.comcast.net) has left #ceph
[23:01] * coredumb (~coredumb@ns.coredumb.net) Quit (Ping timeout: 480 seconds)
[23:02] * dspano (~dspano@rrcs-24-103-221-202.nys.biz.rr.com) Quit (Quit: Leaving)
[23:02] * coredumb (~coredumb@ns.coredumb.net) has joined #ceph
[23:04] * sjust (~sam@2607:f298:a:607:baac:6fff:fe83:5a02) has left #ceph
[23:04] * sjust (~sam@2607:f298:a:607:baac:6fff:fe83:5a02) has joined #ceph
[23:06] * stingray (~stingray@stingr.net) Quit (Quit: shas vse sdoxnet)
[23:09] <PerlStalker> joshd: I was thinking, too, of splitting across data centers then racks using 4 replicas.
[23:09] * noob2 (a5a00214@ircip1.mibbit.com) Quit (Quit: http://www.mibbit.com ajax IRC Client)
[23:10] <joshd> PerlStalker: for that case separate hierarchies for each dc make sense
[23:11] <PerlStalker> joshd: Then I would want a chooseleaf line for dc and one for rack?
[23:17] <elder> joshd, I have been getting features information about every snapshot. Is there any reason I really care about snapshot features unless a snapshot is mapped?
[23:17] <mikeryan> rlr219: that snapset looks right
[23:18] <joshd> PerlStalker: IIRC you'd want a step take dc1, then a chooseleaf 2, emit, and the same for dc2
[23:18] <joshd> elder: no, only the mapped snapshot really matters
[23:19] <PerlStalker> joshd: Ah, I see. The fog is clearing.
[23:21] <elder> joshd, and if a snapshot is mapped, its the features of that snapshot I care about. Whatever the features bits for the base image happen to be shouldn't matter, is that right?
[23:21] <joshd> elder: that's right
[23:21] <elder> I didn't understand that when I wrote that code before.
[23:21] <dmick> they can't really differ right now; presumably they might in the future
[23:22] <elder> But even if they differ, does it matter if it's not mapped?
[23:22] <elder> Or rather, the only one that matters is the mapped one.
[23:22] <dmick> I can't see why. and yes.
[23:22] <joshd> yeah, the only one that matters is the mapped one
[23:23] <elder> Hmm. Well that might simplify things a bit. I'll get the rework I've done already out for review and then may carve out a little stuff that I now realize isn't necessary.
[23:23] <rlr219> mikeryan: ok here goes
[23:24] <PerlStalker> joshd: Thank you, btw, for the help over the last couple of days.
[23:25] <joshd> PerlStalker: you're welcome
[23:28] <mikeryan> rlr219: what kernel version are you on?
[23:30] <rlr219> 3.2.0-29-generic
[23:32] <elder> sagewk, I'm going to get some of my patches committed. How would you prefer I do so? In the past I have put them in the testing branch and move all the extra crap (like btrfs patches) to the end.
[23:32] <sagewk> yeah, that's fine.
[23:32] <sagewk> we can drop the btrfs patches
[23:32] <elder> OK.
[23:32] <elder> I'll do that.
[23:32] <elder> You have a few commits on the end though, I'm assuming those belong.
[23:33] <elder> OK, 4 btrfs commits will be dropped. What about the btrfs patch I got from Josef this week?
[23:35] <rlr219> mikeryan: what that scrub command?
[23:35] <mikeryan> ceph pg scrub 4.2
[23:35] <mikeryan> double check that PG number
[23:36] <rlr219> that's it. here goes
[23:36] * eternaleye_ is now known as eternaleye
[23:36] * eternaleye (~eternaley@tchaikovsky.exherbo.org) Quit (Read error: Connection reset by peer)
[23:38] <rlr219> i guess one thing good is promising is that normally, by this time in the day the osd would have crashed.
[23:38] <rlr219> it is humming through the scrub
[23:38] <mikeryan> cool, that's a good sign
[23:39] <rlr219> brb
[23:42] <rlr219> it crashed, but i think for a different reason
[23:42] * vata (~vata@ Quit (Quit: Leaving.)
[23:48] <rlr219> just sent you the last part via email. I am getting the whole log ready and will upload it ASAP.
[23:49] <rlr219> after this it will have to wait until monday for me, as I am going out of town.
[23:49] <mikeryan> ok, i'll take a look
[23:50] <mikeryan> it most likely crashed on another corrupted set of objects
[23:50] <mikeryan> this is clearly a case of btrfs corruption
[23:50] <mikeryan> yep, that's the same assert fail
[23:51] <rlr219> is there a way to tell what these PGs go too? so maybe we can roll our VM back to a previous snapshot??
[23:51] * eternaleye (~eternaley@tchaikovsky.exherbo.org) has joined #ceph
[23:51] <rlr219> or if it is something we can delete??
[23:51] <mikeryan> the problems are a bit deeper than that
[23:52] <mikeryan> the ceph file store is all around hosed at this point
[23:52] * BManojlovic (~steki@ Quit (Quit: Ja odoh a vi sta 'ocete...)
[23:52] <rlr219> the entire cluster?
[23:53] <mikeryan> this PG, at least
[23:53] <mikeryan> if there was corruption on this PG, it wouldn't be unreasonable to expect corruption on other PGs served by the same OSDs
[23:54] <mikeryan> talking with sjust about potential ways to deal with this
[23:54] <mikeryan> we won't leave you hanging
[23:57] <rlr219> ok. we'll hit it on monday.
[23:57] <mikeryan> alright, sounds good
[23:57] <mikeryan> 1000 apologies that you hit this

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.