#ceph IRC Log

Index

IRC Log for 2013-04-03

Timestamps are in GMT/BST.

[0:00] <andreask> maybe someone can give me some insights ... I already tried to get some information yesterday: using rbd block devices as backend for iscsi target luns -- so the kernel module ... does it work/perform well with multi-threaded access? Any expected/known limitations?
[0:02] <nhm> andreask: if I'm understanding your question right, the kernel client does fine with multi-threaded access. The big limitation is that you do need concurrency to see good performance, either through higher io depth or more clients in general.
[0:06] <nhm> andreask: with 16 volumes I could do an aggregate bandwidth of about 1.5GB/s. In this case this was all coming from the same client. 4MB IOs.
[0:06] <nhm> 24 OSDs, no replication.
[0:07] <andreask> nhm: great, thanks for the valuable information
[0:07] <nhm> np
[0:07] <nhm> ok, off to dinner guys
[0:08] <andreask> bye
[0:08] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[0:08] * loicd (~loic@magenta.dachary.org) has joined #ceph
[0:12] * aliguori (~anthony@cpe-70-112-157-87.austin.res.rr.com) has joined #ceph
[0:12] <pioto> hm, i guess this means i would need to build my own kernel, or something? [105981.075945] libceph: mon2 x.x.x.x:6789 feature set mismatch, my 8a < server's 204008a, missing 2040000
[0:12] <pioto> (when trying to mount cephfs after upgrading to 0.60)
[0:14] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[0:16] * ivotron (~ivo@dhcp-59-232.cse.ucsc.edu) has joined #ceph
[0:16] <gregaf> pioto: ummm, odd…was it mounting correctly with a previous version?
[0:16] <gregaf> the client interface shouldn't have changed there
[0:16] <gregaf> oh, that's for a monitor, not an MDS
[0:16] <gregaf> are you using any of the CRUSH tunables?
[0:17] <gregaf> that's probably what that is
[0:19] <dmick> yep, those are TUNABLES and TUNABLES2
[0:20] <pioto> gregaf: ah, yes
[0:20] <pioto> i had it set to 'optimal'
[0:20] <pioto> guess i want 'bobtail'
[0:20] <gregaf> I'm afraid I don't remember what kernels enable which tunables
[0:21] <pioto> but, well... changing that doesn't seem to help
[0:21] <dmick> optimal is bobtail
[0:21] <pioto> maybe just to 'default' then
[0:21] <gregaf> dmick: I take it you looked up the mapping; were the missing ones all the tunables?
[0:21] <gregaf> in which case he can't use any of them
[0:21] <dmick> the missing ones are 2040000
[0:21] <dmick> that's TUNABLES and TUNABLES2
[0:21] <gregaf> so yeah, no tunables allowed at all
[0:21] <pioto> hm. ok
[0:21] <gregaf> if that kernel needs to be able to communicate
[0:21] <pioto> yeah
[0:22] <gregaf> newer ones enable one or both, forget when they went in (elder?)
[0:22] <pioto> well. hm
[0:22] <pioto> i guess i coulllld use fuse for cephfs there
[0:22] <pioto> but i assume that'd hurt performance a fair bit
[0:22] <dmick> joao, whoever: https://github.com/ceph/ceph/pull/188
[0:24] <gregaf> pioto: depends on what you're doing, but…I doubt it. I've seen some performance deltas in both directions with our clients, and the cost of FUSE is dramatically overstated in a lot of circles
[0:25] <pioto> well. wow. 0.60, rbd cache
[0:25] <pioto> huuuuge win
[0:25] <pioto> so far
[0:25] <pioto> on my mysql benchmarking
[0:25] <pioto> much more consistent throughput
[0:26] <pioto> gregaf: yeah. i don't wanna say FUSE sucks out of hand, because i know tha'ts not the whole story
[0:27] <pioto> but, it still is additional overhead, i think
[0:27] <pioto> anyways... hm
[0:27] <pioto> i guess it's a tradeoff of...
[0:27] <pioto> that overhead vs. the psosible gain from crush tunables
[0:32] * aliguori (~anthony@cpe-70-112-157-87.austin.res.rr.com) Quit (Remote host closed the connection)
[0:39] <nhm> pioto: glad to hear it! The pg_info changes may be helping too if you were using bobtail previously.
[0:40] <pioto> yes, i was on bobtail before
[0:40] <nhm> pioto: how is 0.60 bnechmarking vs bobtail?
[0:40] <pioto> once mysql got to some mass insert tests, though, it seems to have defeated the cache
[0:40] <nhm> pioto: I've done fio tests but not mysql
[0:41] <pioto> but, well, for some of the initial tests, well, it actually finished them
[0:41] <pioto> vs not, over several hours :)
[0:41] <nhm> lol, ok
[0:41] <pioto> so, at least some workloads are still much better
[0:41] <pioto> i'm not seeing anything obvious to tweak for mysql, though...
[0:41] <nhm> pioto: we obviousl have some work to do yet on database performance tuning. :)
[0:41] <pioto> and the ext4 fs used by this guest is using the same 4096 block size as the rbd volume
[0:42] <pioto> so i would guess that for cases where you aren't doing lots of fsyncs(), like i assume mysql does, it probably should work fairly decently
[0:44] <pioto> hm. the rbd cache, though...
[0:44] <pioto> that's per-client
[0:45] <pioto> as in, each vps would have its own allocation for the cache, of the same size?
[0:45] <pioto> hm
[0:46] <nhm> rbd cache is per-client. On the OSD side, journal writes are direct and OSD writes are buffered.
[0:47] <pioto> yeah. i'm not sure if my journal setup is the best
[0:47] <nhm> so data gets appended to the journal sequentially but the data is only written out the OSD as the deemded necessary by the VM layer / scheduler.
[0:47] <pioto> right now, i'm just using the default path (so a file on the OSD filesystem)
[0:47] <pioto> that's on btrfs
[0:47] <pioto> that's using an entire drive
[0:47] <pioto> per osd
[0:48] <nhm> pioto: what kind of controller are you using?
[0:49] <pioto> i think just a standard sata one
[0:49] <pioto> nothing specialized, hardware-wise
[0:53] <andreask> hmm .. how is read-after-write ordering guaranteed if writes go into the journal first but reads are only served from osds?
[0:59] * PerlStalker (~PerlStalk@72.166.192.70) Quit (Quit: ...)
[1:00] <sjust> andreask: both the journal and the filestore are inside of an OSD
[1:00] <sjust> so the osd just delays the read until the prior write hits the filesystem
[1:00] <sjust> put another way: the journal is an implentation detail of an osd
[1:01] <andreask> sjust: ah, makes sense ... thanks for the clarification!
[1:01] <sjust> it's there so that the osd can work with failure atomic transactions
[1:03] * diegows (~diegows@190.190.2.126) Quit (Ping timeout: 480 seconds)
[1:05] * tryggvil (~tryggvil@17-80-126-149.ftth.simafelagid.is) has joined #ceph
[1:10] * tnt (~tnt@91.177.233.80) Quit (Ping timeout: 480 seconds)
[1:11] * BillK (~BillK@124-168-239-213.dyn.iinet.net.au) has joined #ceph
[1:13] * vipr__ (~root@78-23-117-241.access.telenet.be) Quit (Ping timeout: 480 seconds)
[1:15] * ivotron (~ivo@dhcp-59-232.cse.ucsc.edu) Quit (Ping timeout: 480 seconds)
[1:17] <joao> dmick, looks good to me
[1:18] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[1:23] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[1:24] <dmick> joao: k thx
[1:27] * vipr (~root@78-23-117-241.access.telenet.be) has joined #ceph
[1:30] * gentleben (~sseveranc@208.74.182.4.static.etheric.net) Quit (Quit: gentleben)
[1:30] * jlogan2 (~Thunderbi@72.5.59.176) has joined #ceph
[1:31] * jlogan (~Thunderbi@2600:c00:3010:1:943e:a21b:f1f8:c84e) Quit (Ping timeout: 480 seconds)
[1:35] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) Quit (Ping timeout: 480 seconds)
[1:42] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[1:45] * gentleben (~sseveranc@c-98-207-40-73.hsd1.ca.comcast.net) has joined #ceph
[1:45] * rustam (~rustam@94.15.91.30) has joined #ceph
[1:56] * rustam (~rustam@94.15.91.30) Quit (Remote host closed the connection)
[2:07] * mjevans (~mje@209.141.34.79) Quit (Remote host closed the connection)
[2:09] * mjevans (~mje@209.141.34.79) has joined #ceph
[2:11] * dpippenger (~riven@216.103.134.250) Quit (Remote host closed the connection)
[2:20] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[2:20] * loicd (~loic@magenta.dachary.org) has joined #ceph
[2:27] * alram (~alram@38.122.20.226) Quit (Quit: leaving)
[2:28] * ivotron (~ivo@69-170-63-251.static-ip.telepacific.net) has joined #ceph
[2:29] * LeaChim (~LeaChim@02d9ee2d.bb.sky.com) Quit (Read error: Connection reset by peer)
[2:40] * slang1 (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) Quit (Ping timeout: 480 seconds)
[2:43] * houkouonchi-work (~linux@12.248.40.138) Quit (Read error: Connection reset by peer)
[2:44] * houkouonchi-work (~linux@12.248.40.138) has joined #ceph
[2:49] * slang1 (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) has joined #ceph
[2:50] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[3:10] * slang1 (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) Quit (Ping timeout: 480 seconds)
[3:21] * slang1 (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) has joined #ceph
[3:28] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) Quit (Quit: Leaving.)
[3:32] * buck (~buck@bender.soe.ucsc.edu) has left #ceph
[3:45] * jlogan2 (~Thunderbi@72.5.59.176) Quit (Ping timeout: 480 seconds)
[4:12] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) has joined #ceph
[4:14] * yasu` (~yasu`@dhcp-59-149.cse.ucsc.edu) Quit (Remote host closed the connection)
[4:20] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) Quit (Ping timeout: 480 seconds)
[4:26] * chutzpah (~chutz@199.21.234.7) Quit (Quit: Leaving)
[4:32] * ivotron (~ivo@69-170-63-251.static-ip.telepacific.net) Quit (Ping timeout: 480 seconds)
[4:47] * slang1 (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) Quit (Ping timeout: 480 seconds)
[4:58] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) has joined #ceph
[5:03] * Cube (~Cube@12.248.40.138) Quit (Ping timeout: 480 seconds)
[5:09] * Downchuck (62e8084e@ircip3.mibbit.com) has joined #ceph
[5:09] <Downchuck> what's argonaut?
[5:12] <lurbs> The first 'stable' (non-dev) release. Now deprecated.
[5:12] <lurbs> http://en.wikipedia.org/wiki/Ceph_(storage)#History
[5:12] <Downchuck> thanks
[5:12] <Downchuck> I'm starting to become a fan, after many hrs of learning curve.
[5:13] <Downchuck> Any ideas on why radosgw would work with make bucket, list and put, but fail with 400 status on GET?
[5:15] <Downchuck> somehow, even debug rgw = 20 isn't enough to help me on this one
[5:15] * scuttlemonkey (~scuttlemo@fl-184-1-34-163.dhcp.embarqhsd.net) Quit (Ping timeout: 480 seconds)
[5:15] <lurbs> No idea, sorry. I haven't really dealt with RGW.
[5:20] <Downchuck> Looks like it's just a compatibility issue with s3cmd.
[5:21] <Downchuck> Well.. pretty soon, I get to practice failing nodes again. Last time I failed pretty badly not knowing about monmaptool and company.
[5:33] <dmick> Downchuck: a fan eh? cool
[5:33] * themgt (~themgt@24-177-232-181.dhcp.gnvl.sc.charter.com) has joined #ceph
[5:33] * themgt (~themgt@24-177-232-181.dhcp.gnvl.sc.charter.com) Quit (Remote host closed the connection)
[5:34] * themgt (~themgt@24-177-232-181.dhcp.gnvl.sc.charter.com) has joined #ceph
[5:35] * portante (~user@c-24-63-226-65.hsd1.ma.comcast.net) has joined #ceph
[5:38] <Downchuck> Well python boto is working just fine with the gateway, s3cmd is working with everything except "get"
[5:39] <dmick> there are a pile of 400 errors, each with a different S3 error string
[5:40] <dmick> is it a particular failure?
[5:41] <Downchuck> It's a completely vague failure unfortunately.
[5:41] <Downchuck> "ERROR: S3 error: 400 ():"
[5:42] <Downchuck> req 1:0.000245::GET /bucket.domain.com/maintenance.html::http status=400
[5:43] * norbi (~nonline@buerogw01.ispgateway.de) has joined #ceph
[5:43] <Downchuck> could still be very much my fault, "nothing to log for operation"
[5:48] <Downchuck> more tweaks; no luck. regardless of the s3cmd, it does seem like there ought to be better error messages for this instance.
[5:57] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[6:03] <dmick> yeah, that's weird. One would think it would have one of the ones in rgw_html_errors.h.
[6:05] * portante is now known as portante|afl
[6:05] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[6:10] <Downchuck> well, s3-curl is giving me some help, at least it's spitting an error, thanks for the pointer on that file.
[6:15] <Downchuck> https://github.com/ceph/ceph/commit/96896eb092c3b4e0760e56d5228ef0d604951a12
[6:15] <Downchuck> Doh!.
[6:16] <dmick> are you using nginx?
[6:23] <Downchuck> yes.
[6:23] <Downchuck> it made things a little more difficult given the manual.
[6:24] <dmick> if you'd like to contribute setup instructions I'm sure we'd be glad to have them :)
[6:24] <Downchuck> There's finally a patch to skip buffering of output to upstream / fastcgi
[6:25] <Downchuck> sure, I'll put something together this week; I'm doing teardown/rebuilds a few times... so, I can even test my instructions.
[6:25] <dmick> awesome
[6:25] <dmick> that fix should be in the nightly debs, by the way, if you want them to test (i.e. you needn't build them yourself)
[6:25] <dmick> but I gotta run; gl
[6:33] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) Quit (Quit: Leaving.)
[7:03] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) has joined #ceph
[7:16] * wschulze (~wschulze@cpe-69-203-80-81.nyc.res.rr.com) Quit (Quit: Leaving.)
[7:19] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) Quit (Ping timeout: 480 seconds)
[7:24] * Cube (~Cube@cpe-76-172-67-97.socal.res.rr.com) has joined #ceph
[7:50] * wschulze (~wschulze@cpe-69-203-80-81.nyc.res.rr.com) has joined #ceph
[7:50] * tnt (~tnt@91.177.219.219) has joined #ceph
[7:51] * fghaas (~florian@91-119-65-118.dynamic.xdsl-line.inode.at) has joined #ceph
[7:51] * fghaas (~florian@91-119-65-118.dynamic.xdsl-line.inode.at) has left #ceph
[7:53] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[7:55] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[8:10] * ivotron (~ivo@adsl-76-254-17-170.dsl.pltn13.sbcglobal.net) has joined #ceph
[8:15] * wschulze (~wschulze@cpe-69-203-80-81.nyc.res.rr.com) Quit (Quit: Leaving.)
[8:17] * sleinen (~Adium@2001:620:0:25:b4c1:155d:14d1:e913) has joined #ceph
[8:17] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) has joined #ceph
[8:25] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) Quit (Ping timeout: 480 seconds)
[8:41] * madkiss (~madkiss@business-213-023-158-038.static.arcor-ip.net) has joined #ceph
[8:49] * threesome (~threesome@ip-94-113-12-74.net.upcbroadband.cz) Quit (Remote host closed the connection)
[8:52] * sleinen (~Adium@2001:620:0:25:b4c1:155d:14d1:e913) Quit (Quit: Leaving.)
[8:59] * sleinen (~Adium@130.59.94.103) has joined #ceph
[9:01] * sleinen1 (~Adium@2001:620:0:25:61ae:6bfb:ed6b:2b09) has joined #ceph
[9:01] * gerard_dethier (~Thunderbi@85.234.217.115.static.edpnet.net) has joined #ceph
[9:04] * madkiss (~madkiss@business-213-023-158-038.static.arcor-ip.net) Quit (Quit: Leaving.)
[9:04] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[9:05] * madkiss (~madkiss@business-213-023-158-038.static.arcor-ip.net) has joined #ceph
[9:07] * sleinen (~Adium@130.59.94.103) Quit (Ping timeout: 480 seconds)
[9:15] <Downchuck> for nginx users: http://yaoweibin.cn/patches/nginx-1.2.7-no_buffer-v6.patch
[9:16] <Downchuck> and if you're not using v6.x of radosgw; if ($content_length = '') { set $content_length_fixed 0; }
[9:17] * ScOut3R (~ScOut3R@212.96.47.215) has joined #ceph
[9:17] * hybrid512 (~walid@LPoitiers-156-86-25-85.w193-248.abo.wanadoo.fr) has joined #ceph
[9:17] <Downchuck> That said, it looks like radosgw had a range bug anyway, so 0.6x of radosgw really is the way to go.
[9:18] <paravoid> what kind of bug?
[9:20] <Downchuck> "Fix failing > 4MB range requests through radosgw S3 API." https://github.com/ceph/ceph/commit/c83a01d4e8dcd26eec24c020c5b79fcfa4ae44a3
[9:23] <paravoid> ah, thanks
[9:27] * tnt (~tnt@91.177.219.219) Quit (Ping timeout: 480 seconds)
[9:27] * BManojlovic (~steki@91.195.39.5) has joined #ceph
[9:29] * themgt (~themgt@24-177-232-181.dhcp.gnvl.sc.charter.com) Quit (Quit: Pogoapp - http://www.pogoapp.com)
[9:29] * leseb (~Adium@83.167.43.235) has joined #ceph
[9:33] * l0nk (~alex@83.167.43.235) has joined #ceph
[9:36] * tnt (~tnt@212-166-48-236.win.be) has joined #ceph
[9:44] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[9:49] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) has joined #ceph
[9:53] * fghaas (~florian@91-119-65-118.dynamic.xdsl-line.inode.at) has joined #ceph
[10:01] * madkiss (~madkiss@business-213-023-158-038.static.arcor-ip.net) Quit (Quit: Leaving.)
[10:08] * Downchuck (62e8084e@ircip3.mibbit.com) Quit (Quit: http://www.mibbit.com ajax IRC Client)
[10:13] * LeaChim (~LeaChim@02d9ee2d.bb.sky.com) has joined #ceph
[10:13] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[10:14] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[10:19] * tryggvil (~tryggvil@17-80-126-149.ftth.simafelagid.is) Quit (Quit: tryggvil)
[10:26] * loicd (~loic@185.10.252.15) has joined #ceph
[10:28] * stacker100 (~stacker66@152.pool85-58-180.dynamic.orange.es) has joined #ceph
[10:46] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[10:47] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) has joined #ceph
[10:48] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[10:51] * mcclurmc (~mcclurmc@cpc10-cmbg15-2-0-cust205.5-4.cable.virginmedia.com) Quit (Ping timeout: 480 seconds)
[10:57] * dosaboy_ (~dosaboy@host86-161-164-218.range86-161.btcentralplus.com) has joined #ceph
[10:58] * dosaboy_ (~dosaboy@host86-161-164-218.range86-161.btcentralplus.com) Quit ()
[11:00] * dosaboy_alt (~dosaboy_a@host86-161-164-218.range86-161.btcentralplus.com) has joined #ceph
[11:03] * Cube (~Cube@cpe-76-172-67-97.socal.res.rr.com) Quit (Quit: Leaving.)
[11:07] <tnt> Damn, I see there was some nginx talk that I missed... Here I still have an issue with PUT and 100-Continue ...
[11:10] * mattch (~mattch@pcw3047.see.ed.ac.uk) has joined #ceph
[11:29] * The_Bishop (~bishop@2001:470:50b6:0:25dc:e446:71f6:f1b0) Quit (Ping timeout: 480 seconds)
[11:34] * Cube (~Cube@cpe-76-172-67-97.socal.res.rr.com) has joined #ceph
[11:37] * The_Bishop (~bishop@2001:470:50b6:0:e8dc:72bc:e4bf:4533) has joined #ceph
[11:38] * mcclurmc (~mcclurmc@firewall.ctxuk.citrix.com) has joined #ceph
[11:40] * goldfish (~goldfish@91.215.166.4) Quit (Ping timeout: 480 seconds)
[11:45] * Cube (~Cube@cpe-76-172-67-97.socal.res.rr.com) Quit (Ping timeout: 480 seconds)
[11:49] <fghaas> tnt: you do realize you need a modified mod_fastcgi to get support for 100-continue? (I read your comment out of context so there may be some prior discussion that I missed -- sorry for the noise, in that case)
[11:49] * goldfish (~goldfish@91.215.166.4) has joined #ceph
[11:58] * goldfish (~goldfish@91.215.166.4) Quit (Ping timeout: 480 seconds)
[12:00] <tnt> fghaas: well, I'm not using apache. But under lighttpd, I didn't have to do anything. But for nginx I had to use "rgw print continue = false"
[12:04] <dosaboy> I am trying a fresh install of ceph with v0.56. I get an error from mkcephfs as follows:
[12:04] <dosaboy> /sbin/mkcephfs: monitor mon.a has no address defined
[12:04] <dosaboy> I have set the mon addr for mon.a but yet get that error
[12:04] <dosaboy> it worked fine with 0.41
[12:04] <dosaboy> with the same ceph.conf
[12:05] <dosaboy> any odeas?
[12:05] <dosaboy> s/odeas/ideas/
[12:07] * barryo (~borourke@cumberdale.ph.ed.ac.uk) has left #ceph
[12:08] * barryo (~borourke@cumberdale.ph.ed.ac.uk) has joined #ceph
[12:09] <andreask> can you pastebin your config?
[12:09] <joelio> dosaboy: I'd imagine the config options have changed slightly, perhaps you could pastebin a redacted copy of your ceph.conf?
[12:09] <joelio> andreask: snap :)
[12:10] <andreask> ;-)
[12:10] <dosaboy> sure which pastebin do you guys use?
[12:11] <dosaboy> joelio ^
[12:11] <joelio> dosaboy: I use https://gist.github.com/
[12:13] * wschulze (~wschulze@cpe-69-203-80-81.nyc.res.rr.com) has joined #ceph
[12:14] <dosaboy> joelio: https://gist.github.com/anonymous/5299992
[12:15] <dosaboy> description has gotten a bit mangled unfortunately
[12:17] <joelio> can you have letters on mons? Just checking mine and they're numeral
[12:18] <joelio> I think you can actually
[12:18] <joao> you can
[12:18] <dosaboy> quick start says you can ;)
[12:20] <dosaboy> tbh mkcephfs is bring weird because it says in the docs that default config path is /etc/ceph/ceph.conf but if I omit '-c /etc/ceph/ceph.conf' it behaves differenty
[12:20] <dosaboy> even though that is where the conf is
[12:20] <dosaboy> looks buggy to me
[12:21] <joelio> well, initial look over seems ok to me on the ceph.conf side
[12:21] <joelio> hosts definitely accessible?
[12:21] <joelio> ... although if it has worked I'd imagine so
[12:21] <dosaboy> it works fine with 0.41, are there any known issues with mkcephfs in 0.56.4?
[12:22] <joelio> I heard it was being deprecated in favour of another tool iirc.. although I may be wrongf
[12:23] <joelio> I used it in 0.56.x though and it's worked fine everytime for me
[12:23] <joelio> now on 0.60
[12:23] <dosaboy> so what is preferred install method atm?
[12:23] <dosaboy> shoudl I just do everything manually?
[12:25] <joelio> could use chef, juju, ansible or any of the other tools people have used, people on here will be able to guide you. I really get the feeling mkcephfs shoud be working for you though
[12:27] <dosaboy> joelio, afaik the juju charm for ceph uses mkcephfs
[12:27] <dosaboy> I am gonna backtrack, maybe something silly
[12:27] <andreask> dosaboy: and the address is available on the monitor host?
[12:27] <joelio> dosaboy: for reference, here's my working test cluster setup - https://gist.github.com/anonymous/65c3e657cc7afa30001a
[12:28] <dosaboy> thanks for your help guys, I am gonna make sure it is nothing stupid (which it probably is ;))
[12:28] <dosaboy> joelio: awesome thanks
[12:28] * fghaas (~florian@91-119-65-118.dynamic.xdsl-line.inode.at) Quit (Quit: Leaving.)
[12:28] <joelio> n/p hope you get it sorted!
[12:29] <joelio> you may want to move paths about, mine are fairly non-standard (but can't see any issues with them)
[12:34] * loicd (~loic@185.10.252.15) Quit (Ping timeout: 480 seconds)
[12:37] * wschulze (~wschulze@cpe-69-203-80-81.nyc.res.rr.com) has left #ceph
[12:40] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) Quit (Ping timeout: 480 seconds)
[12:42] * elder (~elder@c-71-195-31-37.hsd1.mn.comcast.net) Quit (Read error: Operation timed out)
[12:56] <joelio> /wi2
[12:56] <joelio> doh
[12:58] * diegows (~diegows@190.190.2.126) has joined #ceph
[13:01] * loicd (~loic@3.46-14-84.ripe.coltfrance.com) has joined #ceph
[13:04] * fghaas (~florian@91-119-65-118.dynamic.xdsl-line.inode.at) has joined #ceph
[13:08] * elder (~elder@c-71-195-31-37.hsd1.mn.comcast.net) has joined #ceph
[13:08] * ChanServ sets mode +o elder
[13:10] * ctrl (~ctrl@83.149.8.236) has joined #ceph
[13:11] <ctrl> hi everyone!
[13:11] <ctrl> i have a trouble with speed on rbd image
[13:13] <dosaboy> joelio: I seem to have thing sorted now,not sure what was wrong though. One question about your .conf, I see that yopu have specified a client stanza for RGW client. If I want to specify stanza for RBD client, what directiove would I have to use?
[13:13] <ctrl> im start windows guest in rbd via kvm:libvirt
[13:13] <dosaboy> joelio: [client.rbd]?
[13:13] <ctrl> so, when im write on disk in vm, write speed is about 10 - 15 mb/s
[13:13] <joelio> dosaboy: rbd client? for just logging?
[13:14] <dosaboy> yes
[13:14] <dosaboy> if I specify in [client] everyting logs to the same file
[13:15] <ctrl> any thoughts?
[13:15] <joelio> dosaboy: I don't set that - there are debugging options and logging level changes listed in the docs - http://ceph.com/docs/master/rados/configuration/log-and-debug-ref/
[13:16] <joelio> ctrl: caching on or off?
[13:16] <joelio> dosaboy: that's just logging of course.. for an rbd client you don't need anything in the config... just a copy of the config and a suitable key
[13:16] <dosaboy> jeolio: i guess I would have to only add [client] for rbd client nodes
[13:17] <joelio> no
[13:17] <dosaboy> joelio: does the rbd client not log antyhing?
[13:17] <joelio> minimal iirc - checking now
[13:18] <dosaboy> or does it all go to syslog
[13:18] <joelio> no /var/log/ceph mate
[13:18] <joelio> http://ceph.com/docs/master/rados/operations/debug/
[13:18] <dosaboy> since everything else is configurable I assumed...
[13:18] <joelio> yea, there are sane defaults though :)
[13:20] <dosaboy> joelio: ack
[13:20] <ctrl> joelio: nope, how can i enable caching?
[13:21] * rekby (~Adium@2.92.10.160) has joined #ceph
[13:23] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) has joined #ceph
[13:23] <joelio> ctrl: search is your friend... so is documentation ;) http://ceph.com/docs/master/rbd/rbd-config-ref/
[13:25] <ctrl> joelio: Yeah! Thnx! But in linux vm i have a normal disk speed about 200 - 250 mb/s in write via dd
[13:27] <wido> ctrl: Do you have the RBD cache enabled?
[13:28] <wido> And did you benchmark all the OSDs to see what they are able to write?
[13:28] <wido> And do you have a journal on the OSDs?
[13:28] <wido> Don't be fooled by dd, since it normally does write in writeback mode
[13:28] <ctrl> wido: hi! nope, ceph without rbd cache
[13:29] <wido> So that means that all your writes go to the OSDs directly, which have to replicate it
[13:29] <wido> ctrl: Do you have journals for your OSDs on different disks?
[13:29] <ctrl> wido: yeah, 1 ssd per 6 disks
[13:29] <wido> During the write, how much %util is the SSD?
[13:30] <wido> iostat will tell you that
[13:30] * sleinen1 (~Adium@2001:620:0:25:61ae:6bfb:ed6b:2b09) Quit (Quit: Leaving.)
[13:30] <ctrl> wido: give me a 1 min
[13:30] * sleinen (~Adium@130.59.94.103) has joined #ceph
[13:30] * fghaas (~florian@91-119-65-118.dynamic.xdsl-line.inode.at) Quit (Quit: Leaving.)
[13:33] * sleinen1 (~Adium@130.59.94.103) has joined #ceph
[13:33] * sleinen (~Adium@130.59.94.103) Quit (Read error: Connection reset by peer)
[13:34] * sleinen (~Adium@2001:620:0:26:b991:c8cf:ea7a:5d14) has joined #ceph
[13:36] <ctrl> wido: %util ssd about 10% - 3%
[13:36] <wido> ctrl: What filesystem do you use? btrfs? xfs?
[13:36] <ctrl> wido: xfs
[13:36] <wido> And how many OSDs spread out over how many hosts? Are all OSDs equal in Hardware?
[13:37] <ctrl> wido: well, 12 osd, 1 osd per disk, 6 osd per host
[13:37] <ctrl> wido: only 2 hosts
[13:37] <wido> ctrl: Replication level is 2x or 3x?
[13:37] <ctrl> wido: 2x replica
[13:38] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[13:38] * Cube (~Cube@cpe-76-172-67-97.socal.res.rr.com) has joined #ceph
[13:39] * ctrl2 (~ctrl@83.149.9.173) has joined #ceph
[13:39] <ctrl2> wido: i'm back
[13:39] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[13:41] * rekby (~Adium@2.92.10.160) Quit (Ping timeout: 480 seconds)
[13:41] * sleinen1 (~Adium@130.59.94.103) Quit (Ping timeout: 480 seconds)
[13:45] * ctrl (~ctrl@83.149.8.236) Quit (Ping timeout: 480 seconds)
[13:46] * Cube (~Cube@cpe-76-172-67-97.socal.res.rr.com) Quit (Read error: Operation timed out)
[13:52] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[13:54] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[13:54] <ctrl2> wido: ?
[13:58] * ScOut3R_ (~ScOut3R@212.96.47.215) has joined #ceph
[14:03] * nhorman (~nhorman@hmsreliant.think-freely.org) has joined #ceph
[14:05] * ScOut3R (~ScOut3R@212.96.47.215) Quit (Ping timeout: 480 seconds)
[14:16] <wido> ctrl2: I had to go out for a moment
[14:16] * hflai (~hflai@alumni.cs.nctu.edu.tw) Quit (Read error: Connection reset by peer)
[14:18] <ctrl2> wido: ok
[14:18] <wido> ctrl2: What does the I/O load on the disks do?
[14:23] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) has joined #ceph
[14:23] <ctrl2> wido: only vm's
[14:23] <wido> ctrl2: I mean during the writes
[14:23] * hflai (~hflai@alumni.cs.nctu.edu.tw) has joined #ceph
[14:24] <ctrl2> wido: when i write into rbd?
[14:24] <wido> ctrl2: Yes
[14:25] <ctrl2> wido: i will check it, 2 mins
[14:27] <ctrl2> wido: check i/o load via iostat?
[14:27] <wido> ctrl2: Yes, the %util again
[14:28] * slang1 (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) has joined #ceph
[14:28] * fghaas (~florian@91-119-65-118.dynamic.xdsl-line.inode.at) has joined #ceph
[14:30] <ctrl2> wido: well, ssd is 50 - 60% and disks 60% - 70%
[14:32] <ctrl2> wido: if i write more 2Gb, speed down to 30 - 40mb/s and %util on disk 20% - 10%
[14:33] * sleinen (~Adium@2001:620:0:26:b991:c8cf:ea7a:5d14) Quit (Quit: Leaving.)
[14:33] <wido> ctrl2: I suggest you try adding a rbd cache first (see the docs)
[14:33] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[14:33] <wido> and you can also bench all the OSDs
[14:33] <wido> ceph osd tell 1 bench
[14:33] <wido> and benchmark them all to see their write speeds
[14:35] <ctrl2> wido: bench them? how? :)
[14:35] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[14:35] <wido> ctrl2: With the command I gave
[14:35] <wido> watch "ceph -w" for the output
[14:36] <ctrl2> wido: 36506KB/s wr is the highest
[14:37] <Robe> win 32
[14:38] <ctrl2> wido: ok, after add cache to ceph.conf, i need restart, right?
[14:38] <wido> ctrl2: 36MB/sec is rather slow for a disk, so that could very well be your bottleneck. You have to restart the VM indeed to add the cache
[14:41] <ctrl2> wido: ok, how rbd cache size calculate?
[14:42] <wido> ctrl2: About 32MB or 64MB should be enough
[14:42] <ctrl2> wido: 1 more question plz
[14:42] <ctrl2> wido: 1 ssd for 6 osd is good setup? i think about 1 ssd for 3 osd
[14:43] <wido> ctrl2: It really depends on how fast the SSD is
[14:43] * portante|afl (~user@c-24-63-226-65.hsd1.ma.comcast.net) Quit (Ping timeout: 480 seconds)
[14:43] <wido> I'd recommend at least SATA-600 and I 180GB SSD which you partition to about 25%
[14:44] <ctrl2> wido: seq write speed of ssd 450 - 500 mb/s
[14:45] <ctrl2> wido: well, thx a lot!
[14:45] * aliguori (~anthony@cpe-70-112-157-87.austin.res.rr.com) has joined #ceph
[14:46] <wido> ctrl2: Sequential writes don't tell a lot, but 500MB/sec should be OK. Your mainboard also has to have SATA-600 though
[14:47] * sleinen (~Adium@130.59.94.103) has joined #ceph
[14:49] * sleinen1 (~Adium@2001:620:0:26:5028:97c6:857d:6df0) has joined #ceph
[14:50] <tnt> Mm, my cluster just had a pretty big 'hiccups' with a bunch of rbd vm freezing, had to restart the osds to make things work again. In the OSD log I see dozens of :
[14:50] <tnt> heartbeat_map is_healthy 'OSD::op_tp thread 0x7fc6d4c42700' had timed out after 15
[14:50] <tnt> and also:
[14:50] <tnt> 192.168.0.210:6800/2958 submit_message osd_op_reply(6282904 rb.0.1acd.37cfac28.000000000104 [write 1818624~8192] ondisk = 0) v4 remote, 192.168.0.26:0/2294631700, failed lossy con, dropping message 0x1ca89e00
[14:50] <tnt> when doing a ceph -s I had 'slow requests' and 'stuck peering' PGs ...
[14:52] <fghaas> tnt, does a "ceph pg dump" reveal that the stuck PGs all have one replica in a single OSD?
[14:52] <fghaas> if so, check that OSD for local I/O issues
[14:54] <tnt> fghaas: well, it's running now, restarting the OSDs solved the issue.
[14:54] <fghaas> so your PGs are all unstuck now?
[14:54] <tnt> yup, now it's all fine.
[14:55] * sleinen (~Adium@130.59.94.103) Quit (Ping timeout: 480 seconds)
[15:01] <tnt> I'll try to remember to do a pg dump if it happens again, but being in prod, when it happens the priority is to get stuff working again ASAP.
[15:03] <fghaas> well if it were indeed a local I/O problem, I'd suspect that (i) an OSD restart wouldn't have fixed it, and/or (ii) it would resurface fairly quickly
[15:03] <matt_> any IPoIB users here?
[15:11] <fghaas> matt_: used it before, can't say I'm expert though
[15:11] * yanzheng (~zhyan@101.82.47.238) has joined #ceph
[15:14] <ctrl2> matt_: yeah
[15:16] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[15:17] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[15:21] <matt_> ctrl2, Have you ever had a problem using connected mode?
[15:21] <ctrl2> matt_: no, everyrthing is ok
[15:22] <matt_> I have to run in datagram at the moment otherwise virtual machines using RBD lock up for periods of time
[15:22] <matt_> I think it might just be my switch being a PITA
[15:22] <ctrl2> matt_: any problems in connected mode?
[15:24] * ctrl2 (~ctrl@83.149.9.173) Quit ()
[15:28] * stiller (~Adium@2001:980:87b9:1:7cc2:7ceb:9e0f:228f) has joined #ceph
[15:31] <stiller> Hi, I just realized my .rgw.buckets only has pg_num of 8. I already have about 2,5TB in a bucket. Has increasing this or pg splitting matured to the point where I could use either? Or does this mean a do-over?
[15:35] * eternaleye (~eternaley@tchaikovsky.exherbo.org) Quit (Remote host closed the connection)
[15:35] <fghaas> stiller: does http://comments.gmane.org/gmane.comp.file-systems.ceph.devel/12848 help?
[15:36] <tnt> stiller: I think at worse yu can copy the pool into another pool, delete the original, then rename it.
[15:37] <stiller> tnt: Ah would that work for rgw.buckets now? I read on the mailing list somewhere that it did not use to work.
[15:37] * norbi (~nonline@buerogw01.ispgateway.de) Quit (Quit: Miranda IM! Smaller, Faster, Easier. http://miranda-im.org)
[15:40] * stacker100 (~stacker66@152.pool85-58-180.dynamic.orange.es) Quit (Read error: Operation timed out)
[15:41] <tnt> stiller: not sure. But worth asking on the ML, or trying on a test cluster I guess
[15:42] <stiller> So, to be on the safe side, I would copy the pool to a new one with the correct pg_num set. Then attempt to change pg_num for the pool containing the data. I'll check on the ML to see if changing the pool for rgw currently works.
[15:54] * stacker100 (~stacker66@192.pool85-58-190.dynamic.orange.es) has joined #ceph
[15:55] <dosaboy> if I have user A with 'allow *' for mon and 'allow * pool=foo', should that restrict A to operation on pool foo. I am currenlty able to do 'rbd ls' for any pool using A, is that normal?
[15:57] * yanzheng1 (~zhyan@101.83.207.134) has joined #ceph
[16:00] <fghaas> dosaboy: any change if you change to "allow rwx pool=foo"?
[16:00] <dosaboy> fghaas: lemme check
[16:01] <fghaas> also, do a "ceph auth list" to double check whether there isn't any other permission for that client identity that slipped in
[16:01] <dosaboy> that user has 2 caps; 'caps: [mon] allow *' and 'caps: [osd] allow * pool=images'
[16:01] <dosaboy> thats it
[16:02] <dosaboy> trying rwx...
[16:05] <Azrael> hmm. ok. trying to get ceph 0.60 working on debian squeeze. it has requirements of packages no available, such as leveldb and gdisk and cryptsetup-bin...
[16:05] <Azrael> i've gotten leveldb from the ceph.com/debian-leveldb repo
[16:05] * yanzheng (~zhyan@101.82.47.238) Quit (Ping timeout: 480 seconds)
[16:05] <Azrael> i've gotten gdisk from squeeze-backports
[16:05] <Azrael> but last remaining item is cryptsetup-bin
[16:05] <Azrael> cryptsetup is available in backports but not cryptsetup-bin
[16:05] <Azrael> anybody know where to go from here?
[16:05] <dosaboy> fghaas: is it possible to modify permissions on an existing auth key or do I have to create a new one?
[16:06] * verwilst (~verwilst@110.138-78-194.adsl-static.isp.belgacom.be) has joined #ceph
[16:06] * PerlStalker (~PerlStalk@72.166.192.70) has joined #ceph
[16:08] <fghaas> dosaboy: http://ceph.com/docs/master/rados/operations/auth-intro/ has the ceph-authtool examples you need, I believe
[16:10] <dosaboy> fghaas: I just tried 'sudo ceph-authtool -n client.glance --cap osd "allow rwx pool=images" --cap mon 'allow rwx' /etc/ceph/keyring' but it did not have any effect i.e. client.glance still has *
[16:10] <dosaboy> looks like I have to delete the key and recreate
[16:12] * gucki (~smuxi@77-56-36-164.dclient.hispeed.ch) has joined #ceph
[16:12] <dosaboy> also, the 'ceph auth get-or-create-key' command mentioned in numerous places at ceph.com is invalid
[16:12] <dosaboy> unless I am misreading it ;)
[16:13] <gucki> hi all. is it safe to upgrade from 0.56.3 to 0.56.4 or have problems been reported during the past days? :)
[16:13] <dosaboy> ceph auth appears to support add|del|list|get
[16:14] <dosaboy> although get is not mentined in he docs
[16:14] <fghaas> ceph auth get-or-create-key works just dandy for me; which version are you on?
[16:15] <fghaas> and it's nicely explained in "ceph --help" too
[16:19] <Azrael> ok fuck it. wheezy it is.
[16:19] <Azrael> sup fghaas, hows it going
[16:19] <fghaas> very well, thanks! yourself?
[16:20] <Azrael> perty good, perty good
[16:20] <Azrael> i'm enjoying the extra daylight thanks to dst
[16:20] <Azrael> can't wait for the famous 11pm+ sunsets
[16:23] * sdx23 (~sdx23@with-eyes.net) Quit (Ping timeout: 480 seconds)
[16:23] <dosaboy> fghaas: oops my bad, that was for a really old version of ceph
[16:23] <dosaboy> I'm playing with a few versions at once here ;)
[16:24] <Azrael> fghaas: are you going to oscon this year?
[16:24] <fghaas> Azrael: nope, I already ranted here yesterday about lack of ceph talks there
[16:25] <Azrael> ahh ok. yeah i dont think i'll be there this year either.
[16:25] <fghaas> but I will be in that same convention center (and at some point, likely the burger place next to it) week after next
[16:25] <Azrael> for what conference?
[16:26] <fghaas> openstack summit
[16:26] <Azrael> oh nice
[16:26] <Azrael> thats the developer summit yeah?
[16:26] <fghaas> that's the design summit and user conference rolled into one, yes
[16:26] <Azrael> nice
[16:27] <Azrael> out of curiosity, have you come across m/any openstack deployments in production with (xen|xcp|xenserver) besides rackspace?
[16:30] <fghaas> most folks seem to prefer libvirt/kvm, that's for sure :)
[16:30] <fghaas> but yeah, rackspace is a pretty big force to reckon with in the openstack/xen space
[16:30] <matt_> argh, this monitor bug is killing me
[16:31] <absynth> which one?
[16:32] <matt_> http://tracker.ceph.com/issues/4521
[16:32] <matt_> I currently can't start or restart OSD's
[16:38] <absynth> how lovely
[16:39] <matt_> that'll learn me not to upgrade without checking the bug tracker...
[16:40] <absynth> well, at least you got "priority: urgent"
[16:41] <matt_> I'm sure the inktank guys will get it all sorted
[16:41] <matt_> I'm just sulking :)
[16:41] * morse (~morse@supercomputing.univpm.it) Quit (Remote host closed the connection)
[16:42] * morse (~morse@supercomputing.univpm.it) has joined #ceph
[16:43] * rekby (~Adium@2.92.10.160) has joined #ceph
[16:45] <rekby> Hello.
[16:46] <rekby> I try ceph-cluster, now I have stable state:
[16:47] <rekby> ceph -s
[16:47] <rekby> health HEALTH_WARN 198 pgs degraded; 198 pgs stuck unclean
[16:47] <rekby> monmap e1: 1 mons at {a=81.177.175.254:6789/0}, election epoch 1, quorum 0 a
[16:47] <rekby> osdmap e72: 3 osds: 3 up, 3 in
[16:47] <rekby> pgmap v5419: 768 pgs: 570 active+clean, 198 active+degraded; 2044 MB data, 7045 MB used, 203 GB / 209 GB avail
[16:47] <rekby> mdsmap e16: 1/1/1 up {0=a=up:active}
[16:47] <rekby> I try http://eu.ceph.com/docs/wip-3060/ops/manage/failures/osd/#failures-osd-unfound but ceph don't find any missed object
[16:49] <rekby> how I can repair cluster?
[16:54] <matt_> rekby, did you lose some osd's?
[16:54] <rekby> no
[16:54] <rekby> hello, matt_
[16:55] <matt_> are all of the osd's on the same host or two hosts?
[16:55] <rekby> I try it:
[16:55] <rekby> 1. I have OK cluster with 3 osd
[16:55] <rekby> 2. I set pool size 3 for all 3 pools. (data, metadata, rbd)
[16:55] <rekby> 3. Turn off 2 osd
[16:55] <rekby> 4. set pool size 2 for all pools
[16:55] <rekby> 5. turn on all osd
[16:56] <rekby> 3 osd on 3 different host
[16:56] <matt_> Try restarting all of the osd's if you haven't already
[16:57] <joao> matt_, I'm on that bug now
[16:57] <joao> been looking into it this week, but I think it should be sorted out today still
[16:57] <matt_> joao, you are a legend. Thank you
[16:59] <rekby> matt, i tried: service ceph -a restart,
[16:59] <rekby> ceph osd repair *
[16:59] <rekby> ceph osd scrub 0
[16:59] <rekby> ceph osd scrub 1
[16:59] <rekby> ceph osd scrub 2
[17:01] <rekby> hmm... No, i don't tried restart - I try in now
[17:02] * jlogan1 (~Thunderbi@2600:c00:3010:1:64ea:852f:5756:f4bf) has joined #ceph
[17:02] <rekby> restart cluster is help me, thanks.
[17:03] <matt_> rekby, Good to hear. Sometimes OSD's just get stuck, a restart usually fixes the problem
[17:05] <rekby> Where I can see help by all commands? for example
[17:05] <rekby> on page http://eu.ceph.com/docs/wip-3060/ops/manage/failures/osd/#failures-osd-unfound
[17:05] <rekby> i see command ceph pg 2.4 list_missing, but I don't see it on page http://ceph.com/docs/master/rados/operations/control/#placement-group-subsystem
[17:05] <rekby> on anywere else
[17:05] <fghaas> rekby: ceph --help
[17:07] <rekby> mm, it is good, i tried only man ceph, thanks
[17:07] <rekby> but ceph —help don't show all commanmd, for example list_missing
[17:09] * JohansGlock (~quassel@kantoor.transip.nl) Quit (Remote host closed the connection)
[17:12] * gerard_dethier (~Thunderbi@85.234.217.115.static.edpnet.net) Quit (Quit: gerard_dethier)
[17:16] * BMDan (~BMDan@static-108-44-155-130.clppva.fios.verizon.net) has joined #ceph
[17:19] * verwilst (~verwilst@110.138-78-194.adsl-static.isp.belgacom.be) Quit (Quit: Ex-Chat)
[17:21] * JohansGlock (~quassel@kantoor.transip.nl) has joined #ceph
[17:25] * BManojlovic (~steki@91.195.39.5) Quit (Quit: Ja odoh a vi sta 'ocete...)
[17:25] * Vjarjadian (~IceChat77@5ad6d005.bb.sky.com) has joined #ceph
[17:35] * yanzheng1 (~zhyan@101.83.207.134) Quit (Ping timeout: 480 seconds)
[17:50] * ScOut3R_ (~ScOut3R@212.96.47.215) Quit (Ping timeout: 480 seconds)
[18:02] * gucki (~smuxi@77-56-36-164.dclient.hispeed.ch) Quit (Remote host closed the connection)
[18:07] * hflai (~hflai@alumni.cs.nctu.edu.tw) Quit (Remote host closed the connection)
[18:08] * BillK (~BillK@124-168-239-213.dyn.iinet.net.au) Quit (Ping timeout: 480 seconds)
[18:09] * alram (~alram@38.122.20.226) has joined #ceph
[18:10] * sleinen1 (~Adium@2001:620:0:26:5028:97c6:857d:6df0) Quit (Quit: Leaving.)
[18:11] * sleinen (~Adium@130.59.94.103) has joined #ceph
[18:11] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) Quit (Ping timeout: 480 seconds)
[18:12] * hflai (~hflai@alumni.cs.nctu.edu.tw) has joined #ceph
[18:19] * sleinen (~Adium@130.59.94.103) Quit (Ping timeout: 480 seconds)
[18:21] * BillK (~BillK@58-7-53-67.dyn.iinet.net.au) has joined #ceph
[18:36] * slang1 (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) Quit (Quit: Leaving.)
[18:36] * tnt (~tnt@212-166-48-236.win.be) Quit (Ping timeout: 480 seconds)
[18:37] * fghaas (~florian@91-119-65-118.dynamic.xdsl-line.inode.at) has left #ceph
[18:37] * slang (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) has joined #ceph
[18:37] * Mastema (~luci@pool-173-48-36-205.bstnma.fios.verizon.net) Quit (Read error: Operation timed out)
[18:38] * rahmu (~rahmu@83.167.43.235) has joined #ceph
[18:38] * eschnou (~eschnou@191.208-201-80.adsl-dyn.isp.belgacom.be) has joined #ceph
[18:38] * Mastema (~luci@pool-173-48-36-205.bstnma.fios.verizon.net) has joined #ceph
[18:39] * sleinen (~Adium@217-162-132-182.dynamic.hispeed.ch) has joined #ceph
[18:40] * sleinen1 (~Adium@2001:620:0:25:58d5:9813:f8a4:5c3) has joined #ceph
[18:43] * l0nk (~alex@83.167.43.235) Quit (Quit: Leaving.)
[18:46] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) has joined #ceph
[18:47] * sleinen (~Adium@217-162-132-182.dynamic.hispeed.ch) Quit (Ping timeout: 480 seconds)
[18:48] * leseb (~Adium@83.167.43.235) Quit (Quit: Leaving.)
[18:54] * eternaleye (~eternaley@tchaikovsky.exherbo.org) has joined #ceph
[19:03] * sleinen1 (~Adium@2001:620:0:25:58d5:9813:f8a4:5c3) Quit (Quit: Leaving.)
[19:04] * sjustlaptop (~sam@71-83-191-116.dhcp.gldl.ca.charter.com) has joined #ceph
[19:08] * goldfish (~goldfish@91.215.166.4) has joined #ceph
[19:08] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) Quit (Ping timeout: 480 seconds)
[19:10] * chutzpah (~chutz@199.21.234.7) has joined #ceph
[19:11] * gentleben (~sseveranc@c-98-207-40-73.hsd1.ca.comcast.net) Quit (Quit: gentleben)
[19:11] * BillK (~BillK@58-7-53-67.dyn.iinet.net.au) Quit (Read error: Connection reset by peer)
[19:11] * ivotron (~ivo@adsl-76-254-17-170.dsl.pltn13.sbcglobal.net) Quit (Ping timeout: 480 seconds)
[19:14] <gregaf> joao: so what's up with the monitor timeouts? I've been following your email thread but am not sure where you ended up as you seemed to have several ideas that didn't pan out :)
[19:15] * eschnou (~eschnou@191.208-201-80.adsl-dyn.isp.belgacom.be) Quit (Ping timeout: 480 seconds)
[19:17] * sleinen (~Adium@217-162-132-182.dynamic.hispeed.ch) has joined #ceph
[19:18] * yehudasa (~yehudasa@2607:f298:a:607:9911:7a60:6fef:7eb5) Quit (Ping timeout: 480 seconds)
[19:20] * rahmu (~rahmu@83.167.43.235) Quit (Remote host closed the connection)
[19:23] * scuttlemonkey (~scuttlemo@fl-184-1-34-163.dhcp.embarqhsd.net) has joined #ceph
[19:23] * ChanServ sets mode +o scuttlemonkey
[19:25] * sleinen (~Adium@217-162-132-182.dynamic.hispeed.ch) Quit (Ping timeout: 480 seconds)
[19:26] * sleinen (~Adium@2001:620:0:25:9918:629d:5d81:a5a6) has joined #ceph
[19:26] * yehudasa (~yehudasa@2607:f298:a:607:2981:dee6:99f7:e854) has joined #ceph
[19:27] * tnt (~tnt@ptra-178-50-78-38.mobistar.be) has joined #ceph
[19:27] * sleinen1 (~Adium@217-162-132-182.dynamic.hispeed.ch) has joined #ceph
[19:28] * sleinen2 (~Adium@2001:620:0:25:54e4:1cf:2b82:929f) has joined #ceph
[19:28] * KindTwo (~KindOne@h36.177.130.174.dynamic.ip.windstream.net) has joined #ceph
[19:29] <rekby> I try to resize rbd:
[19:29] <rekby> [root@localhost ~]# rbd create t1 --size 10
[19:29] <rekby> [root@localhost ~]# rbd resize t1 --size 20
[19:29] <rekby> 2013-04-03 21:29:15.452130 7f30e9768760 -1 librbd: Error listing snapshots: (95) Operation not supported
[19:29] <rekby> rbd: error opening image t1: (95) Operation not supported
[19:30] <dmick> are all your OSDs running the same version, and which version is that?
[19:30] <rekby> I found only http://www.spinics.net/lists/ceph-users/msg00218.html
[19:30] <rekby> for i in $(ceph osd ls); do ceph tell osd.$i version; done
[19:30] <rekby> ceph version 0.56.3 (6eb7e15a4783b122e9b0c85ea9ba064145958aa5)
[19:30] <rekby> ceph version 0.56.3 (6eb7e15a4783b122e9b0c85ea9ba064145958aa5)
[19:30] <rekby> ceph version 0.56.3 (6eb7e15a4783b122e9b0c85ea9ba064145958aa5)
[19:30] * fghaas (~florian@91-119-65-118.dynamic.xdsl-line.inode.at) has joined #ceph
[19:30] * fghaas (~florian@91-119-65-118.dynamic.xdsl-line.inode.at) has left #ceph
[19:30] <dmick> lol, that was the quickest answer ever!
[19:30] <rekby> ls /usr/lib/rados-classes
[19:30] <rekby> ls: cannot access /usr/lib/rados-classes: No such file or directory
[19:30] <dmick> that would be the problem
[19:31] <dmick> did you build and install by hand?
[19:31] <janos> i dont have that eitehr - installed via yum
[19:31] <rekby> no, I install ceph from epel repository
[19:31] * BillK (~BillK@58-7-57-190.dyn.iinet.net.au) has joined #ceph
[19:31] <alexxy[home]> hi all!
[19:32] <alexxy[home]> i have problems with mds on ceph 0.60
[19:32] <dmick> then that's a repo bug. janos, rekby: can you give me exact details of which repo you used?
[19:32] <janos> same version too
[19:32] <janos> sure thing
[19:32] <janos> from ceph site instructions but lemme get the repo file
[19:32] * sleinen2 (~Adium@2001:620:0:25:54e4:1cf:2b82:929f) has left #ceph
[19:33] <tchmnkyz-> so i have a osd that is down completely and i need to remove it from the cluster but it wont let me because ceph still thinks it is up. is there a way to force it down?
[19:33] * KindOne (KindOne@0001a7db.user.oftc.net) Quit (Ping timeout: 480 seconds)
[19:33] <janos> [ceph]
[19:33] <janos> name=Ceph
[19:33] <janos> baseurl=http://ceph.com/rpms/fc18/$basearch
[19:33] <janos> enabled=1
[19:33] <janos> gpgcheck=1
[19:33] <janos> gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CEPH
[19:33] <rekby> I haму clean centos 6.4, than
[19:33] <rekby> rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
[19:33] * loicd (~loic@3.46-14-84.ripe.coltfrance.com) Quit (Quit: Leaving.)
[19:33] <joao> gregaf, what do you mean?
[19:33] <janos> that's the content of my ceph.repo file. i got the rpm initially from ceph.com
[19:34] * sleinen (~Adium@2001:620:0:25:9918:629d:5d81:a5a6) Quit (Ping timeout: 480 seconds)
[19:34] <alexxy[home]> log with mds crash
[19:34] <alexxy[home]> http://bpaste.net/show/88723/
[19:34] <tchmnkyz-> nm found it
[19:34] <janos> dmick - been using these instructions - http://ceph.com/docs/master/install/rpm/
[19:34] * KindOne (~KindOne@0001a7db.user.oftc.net) has joined #ceph
[19:34] <joao> gregaf, just saw that thread, been away most of the afternoon, and Jim's issue with the paxos, as we found out yesterday, was some (way too) verbose output during Paxos::begin()
[19:34] <janos> modified to suit fedora18
[19:35] <rekby> sorry, I had clean centos 6.3
[19:35] * sleinen1 (~Adium@217-162-132-182.dynamic.hispeed.ch) Quit (Ping timeout: 480 seconds)
[19:35] <janos> here we are - from here - http://ceph.com/rpms/fc18/x86_64/ceph-release-1-0.fc18.noarch.rpm
[19:35] <dmick> janos, rekby: on Debian that dir is part of the 'ceph' pkg
[19:36] <dmick> if you show those contents, nothing in /usr/lib/rados-classes appears?
[19:36] <rekby> dmick, I use ceph without auth
[19:36] <dmick> doesn't matter
[19:36] <janos> same
[19:36] <gregaf> joao: ah, I thought that even with the cleanups it was still taking forever for some unknown reason
[19:36] <gregaf> if it's just the debug output then I'm a very happy camper :)
[19:37] <alexxy[home]> sage: ping
[19:37] <janos> dmick - not sure what you mean by "nothing appears". do you mean i need to get into the rpm itself to inspect?
[19:37] * KindTwo (~KindOne@h36.177.130.174.dynamic.ip.windstream.net) Quit (Ping timeout: 480 seconds)
[19:37] <rekby> I have /usr/lib64/rados-classes
[19:37] <rekby> ls /usr/lib64/rados-classes
[19:37] <rekby> libcls_kvs.so libcls_kvs.so.1.0.0 libcls_lock.so.1 libcls_rbd.so.1 libcls_refcount.so libcls_refcount.so.1.0.0 libcls_rgw.so.1
[19:37] <rekby> libcls_kvs.so.1 libcls_lock.so libcls_lock.so.1.0.0 libcls_rbd.so.1.0.0 libcls_refcount.so.1 libcls_rgw.so libcls_rgw.so.1.0.0
[19:38] * mcclurmc (~mcclurmc@firewall.ctxuk.citrix.com) Quit (Ping timeout: 480 seconds)
[19:38] <janos> yep, same. have lib64
[19:38] <dmick> janos: I'm not sure how rpm querying works. I'm asking if you can see the contents of the ceph package on your system and see if there's a rados-classes installed
[19:38] <dmick> and .. hm.
[19:38] <dmick> ok
[19:39] <janos> brb, something for $work calls. real quick
[19:41] <dmick> rekby: can you tell me what ceph --admin-daemon /var/lib/ceph/osd/ceph-osd.asok config dump | grep rados-classes shows?
[19:41] <dmick> (assuming the default location of the admin socket)
[19:41] <dmick> and, sorry, it's not dump, it's show
[19:41] <dmick> let's try that again
[19:42] <dmick> ceph --admin-daemon /var/lib/ceph/osd/ceph-osd.asok config show | grep rados-classes
[19:42] <janos> back
[19:43] <janos> i get not found. hunting for it
[19:43] <dmick> how about grep for osd_class_dir
[19:43] <dmick> (or you mean the admin socket)
[19:43] <janos> admin socket
[19:44] <dmick> *argh* that's because I got it wrong again
[19:44] <dmick> I'm sorry
[19:44] <rekby> ceph --admin-daemon /var/lib/ceph/osd/ceph-osd.asok config show | grep rados-classes
[19:44] <janos> np
[19:44] <rekby> connect to /var/lib/ceph/osd/ceph-osd.asok failed with (2) No such file or directory
[19:44] <rekby> ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show | grep rados-classes
[19:44] <rekby> "osd_class_dir": "\/usr\/lib64\/rados-classes",
[19:44] <joao> gregaf, the issue Jim is now hitting appears to be different from the one he hit yesterday
[19:44] <dmick> /var/run/ceph/$cluster-$name.asok is the default
[19:44] <dmick> rekby: ok, then that's odd, because that ought to be working
[19:44] <joao> gregaf, looks like it's basically what Sage suggested: an overloaded monitor becoming increasingly overloaded due to being already overloaded
[19:44] <gregaf> yeah
[19:45] <gregaf> I just assumed that issue was triggered by the slow commits
[19:45] <gregaf> has he confirmed his PGMap commits are taking reasonable amounts of time now?
[19:45] <janos> dmick i did ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show | grep rados-classes
[19:45] <janos> and got :
[19:45] <janos> "osd_class_dir": "\/usr\/lib64\/rados-classes",
[19:46] <joao> yeah, last email he sent yesterday on the paxos thread showed normal commit times
[19:46] <janos> that looks right to me. but i'm not terribly knowledgable
[19:46] <dmick> janos: do you have the same problem (resize doesn't work)?
[19:46] <joao> gregaf, most of his time was being spent outputting a list of pending proposals, which has a debug level of 10 instead of 30 (not even being there would be even better)
[19:46] <janos> i have not tried that on 0.56.3
[19:47] <janos> i can see. just make a pool and try to resize it?
[19:47] <dmick> not pool, image
[19:47] <gregaf> joao: yeah, sounds good then; I was just out of date :)
[19:47] <janos> ah rbd. yes, scrolled back
[19:48] <dmick> $ rbd create -s 1 image
[19:48] <dmick> $ rbd resize -s 2 image
[19:48] <janos> Resizing image: 100% complete...done.
[19:49] <dmick> ok, so it's something more subtle than finding the class for rekby
[19:49] * stacker100 (~stacker66@192.pool85-58-190.dynamic.orange.es) Quit (Ping timeout: 480 seconds)
[19:49] * tnt (~tnt@ptra-178-50-78-38.mobistar.be) Quit (Ping timeout: 480 seconds)
[19:49] <janos> removing that image worked fine too
[19:51] <rekby> dmick - it is my first, test cluster on empty servers - I can do/add/remove anything
[19:52] <janos> mine is actually holding things i like!
[19:52] <janos> ;)
[19:52] <dmick> yeah, trying to figure out which debug flag will help
[19:53] * gentleben (~sseveranc@208.74.182.4.static.etheric.net) has joined #ceph
[19:53] * dosaboy1 (~gizmo@host86-161-164-218.range86-161.btcentralplus.com) has joined #ceph
[19:54] * dosaboy1 (~gizmo@host86-161-164-218.range86-161.btcentralplus.com) Quit ()
[19:54] * calebamiles (~caleb@c-50-138-218-203.hsd1.vt.comcast.net) Quit (Read error: Operation timed out)
[19:54] * dosaboy_alt (~dosaboy_a@host86-161-164-218.range86-161.btcentralplus.com) Quit (Quit: leaving)
[19:55] * dosaboy_alt (~dosaboy_a@host86-161-164-218.range86-161.btcentralplus.com) has joined #ceph
[19:58] * dosaboy_alt (~dosaboy_a@host86-161-164-218.range86-161.btcentralplus.com) Quit ()
[20:03] * dpippenger (~riven@216.103.134.250) has joined #ceph
[20:03] <dmick> ok rekby
[20:04] <rekby> yes, i here, waiting
[20:04] <dmick> 1) find out where the object in question lives: ceph osd map rbd image.rbd
[20:04] <dmick> where 'rbd' is the pool you put the image in (rbd by default, which I assume)
[20:04] <dmick> and 'image' is the name of your image
[20:05] <dmick> that should show you an 'up' set and an 'acting' set which ought to be the same. First number is the primary OSD for that object (which is the rbd image header)
[20:05] <dmick> and that's the one will use to turn on debugging:
[20:05] <dmick> ceph tell osd.<n> injectargs --debug_objclass=30
[20:05] <dmick> which should return debug_objclass=30/30
[20:06] <dmick> then, try the resize
[20:06] <dmick> and look in the OSD log for messages mentioning 'cls'
[20:07] <rekby> ceph osd map rbd test.rdb
[20:07] <rekby> osdmap e103 pool 'rbd' (2) object 'test.rdb' -> pg 2.d46a7337 (2.37) -> up [2,1] acting [2,1]
[20:07] <dmick> rbd
[20:07] <dmick> not rdb
[20:07] <rekby> ceph osd map rbd test.rbd
[20:07] <rekby> osdmap e103 pool 'rbd' (2) object 'test.rbd' -> pg 2.a2fa4514 (2.14) -> up [1,0] acting [1,0]
[20:08] <dmick> so its primary is osd 1, and that's the one you want to set the debug flag on
[20:08] <dmick> actually just for good measure, let's set two flags
[20:08] <rekby> my image have name test.
[20:08] <rekby> Do I must execute "ceph osd map rbd test" or "ceph osd map rbd test.rbd"?
[20:08] <dmick> ceph tell osd.1 injectargs --debug_objclass=30
[20:09] <dmick> ceph tell osd.1 injectargs --debug_osd=30
[20:09] <dmick> test.rbd is the object that holds the header info for rbd image 'test'
[20:09] <dmick> (rbd images are made up of several objects: a header obj and several data objects, plus some common objects holding directory info)
[20:10] * calebamiles (~caleb@c-50-138-218-203.hsd1.vt.comcast.net) has joined #ceph
[20:12] <rekby> ok
[20:12] <rekby> [root@localhost ceph]# ceph tell osd.1 injectargs --debug_objclass=30
[20:12] <rekby> ignoring empty injectargs
[20:12] <rekby> [root@localhost ceph]# ceph tell osd.1 injectargs --debug_osd=30
[20:12] <rekby> ignoring empty injectargs
[20:12] <rekby> 2013-04-03 22:10:14.231164 7f5c729f7700 0 osd.1 103 do_command r=-22 ignoring empty injectargs
[20:12] <rekby> 2013-04-03 22:10:14.231210 7f5c729f7700 0 log [INF] : ignoring empty injectargs
[20:12] <rekby> 2013-04-03 22:10:33.429936 7f5c729f7700 0 osd.1 103 do_command r=-22 ignoring empty injectargs
[20:12] <rekby> 2013-04-03 22:10:33.429948 7f5c729f7700 0 log [INF] : ignoring empty injectargs
[20:13] <dmick> try quoting that last arg
[20:14] <rekby> [root@localhost ~]# ceph tell osd.1 injectargs --debug_objclass="30"
[20:14] <rekby> ignoring empty injectargs
[20:14] <rekby> [root@localhost ~]# ceph tell osd.1 injectargs "--debug_objclass=30"
[20:14] <rekby> ignoring empty injectargs
[20:14] <dmick> grrr
[20:17] <rekby> ceph tell osd.1 injectargs "--debug_objclass 30"
[20:17] <dmick> that worked? without the =?
[20:18] <rekby> yes, i found it on http://ceph.com/docs/master/rados/operations/control/
[20:18] <dmick> I don't know why that's different, but ok
[20:21] <rekby> 2013-04-03 22:20:45.044525 7f5c84213700 5 osd.1 103 tick
[20:21] <rekby> 2013-04-03 22:20:45.044674 7f5c84213700 20 osd.1 103 scrub_should_schedule loadavg 0 < max 0.5 = yes
[20:21] <rekby> 2013-04-03 22:20:45.044699 7f5c84213700 20 osd.1 103 sched_scrub load_is_low=1
[20:21] <rekby> 2013-04-03 22:20:45.044705 7f5c84213700 30 osd.1 103 0.c at 2013-04-03 15:02:18.563768
[20:21] <rekby> 2013-04-03 22:20:45.044735 7f5c84213700 10 osd.1 103 0.c at 2013-04-03 15:02:18.563768 > min 2013-04-02 22:20:45.044703 (86400 seconds ago)
[20:21] <rekby> 2013-04-03 22:20:45.044750 7f5c84213700 20 osd.1 103 sched_scrub done
[20:21] <rekby> 2013-04-03 22:20:45.044756 7f5c84213700 25 osd.1 103 heartbeat_check osd.0 first_tx 2013-04-03 21:23:58.288346 last_tx 2013-04-03 22:20:40.952041 last_rx 2013-04-03 22:20:40.952041
[20:21] <rekby> 2013-04-03 22:20:45.044766 7f5c84213700 25 osd.1 103 heartbeat_check osd.2 first_tx 2013-04-03 21:24:11.189165 last_tx 2013-04-03 22:20:40.952041 last_rx 2013-04-03 22:20:40.952041
[20:21] <rekby> 2013-04-03 22:20:46.044926 7f5c84213700 5 osd.1 103 tick
[20:21] <rekby> 2013-04-03 22:20:46.044968 7f5c84213700 20 osd.1 103 scrub_random_backoff lost coin flip, randomly backing off
[20:21] <rekby> 2013-04-03 22:20:46.044975 7f5c84213700 25 osd.1 103 heartbeat_check osd.0 first_tx 2013-04-03 21:23:58.288346 last_tx 2013-04-03 22:20:40.952041 last_rx 2013-04-03 22:20:40.952041
[20:21] <rekby> 2013-04-03 22:20:46.045000 7f5c84213700 25 osd.1 103 heartbeat_check osd.2 first_tx 2013-04-03 21:24:11.189165 last_tx 2013-04-03 22:20:40.952041 last_rx 2013-04-03 22:20:40.952041
[20:21] <rekby> 2013-04-03 22:20:46.852269 7f5c71ff6700 30 osd.1 103 heartbeat_entry woke up
[20:21] <rekby> 2013-04-03 22:20:46.852296 7f5c71ff6700 30 osd.1 103 heartbeat
[20:21] <rekby> 2013-04-03 22:20:46.852359 7f5c71ff6700 30 osd.1 103 heartbeat checking stats
[20:21] <rekby> 2013-04-03 22:20:46.852374 7f5c71ff6700 20 osd.1 103 update_osd_stat osd_stat(2248 MB used, 69389 MB avail, 71637 MB total, peers [0,2]/[])
[20:21] <dmick> yikes no stop
[20:21] <rekby> 2013-04-03 22:20:46.852382 7f5c71ff6700 5 osd.1 103 heartbeat: osd_stat(2248 MB used, 69389 MB avail, 71637 MB total, peers [0,2]/[])
[20:21] <rekby> 2013-04-03 22:20:46.852388 7f5c71ff6700 30 osd.1 103 heartbeat allocating ping for osd.0
[20:21] <rekby> 2013-04-03 22:20:46.852392 7f5c71ff6700 30 osd.1 103 heartbeat sending ping to osd.0
[20:21] <rekby> 2013-04-03 22:20:46.852443 7f5c71ff6700 30 osd.1 103 heartbeat allocating ping for osd.2
[20:21] <rekby> 2013-04-03 22:20:46.852446 7f5c71ff6700 30 osd.1 103 heartbeat sending ping to osd.2
[20:21] <rekby> 2013-04-03 22:20:46.852461 7f5c71ff6700 30 osd.1 103 heartbeat check
[20:21] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) has joined #ceph
[20:21] <rekby> 2013-04-03 22:20:46.852467 7f5c71ff6700 25 osd.1 103 heartbeat_check osd.0 first_tx 2013-04-03 21:23:58.288346 last_tx 2013-04-03 22:20:46.852386 last_rx 2013-04-03 22:20:40.952041
[20:21] <rekby> 2013-04-03 22:20:46.852475 7f5c71ff6700 25 osd.1 103 heartbeat_check osd.2 first_tx 2013-04-03 21:24:11.189165 last_tx 2013-04-03 22:20:46.852386 last_rx 2013-04-03 22:20:40.952041
[20:21] <rekby> 2013-04-03 22:20:46.852483 7f5c71ff6700 30 osd.1 103 heartbeat lonely?
[20:21] <rekby> 2013-04-03 22:20:46.852485 7f5c71ff6700 30 osd.1 103 heartbeat done
[20:22] <gregaf> dmick: injectargs goes through some different parsing paths than startup config options, which impose some weird checks in various places
[20:22] <gregaf> (it's not great, no)
[20:22] <dmick> 1) pastebin.com
[20:22] <dmick> gregaf: I know, but = works on my test cluster and I don't see any recent changes so I'm flummoxed
[20:23] * BillK (~BillK@58-7-57-190.dyn.iinet.net.au) Quit (Read error: Operation timed out)
[20:23] <dmick> 2) rekby: open that log with less or something and look for 'class'; I'm expecting to find some error about loading the class
[20:23] <dmick> if you find some suspicious looking lines and want to show more than one or two, use pastebin.com
[20:24] <joshd> dmick: rekby: you want ceph tell osd.1 injectargs -- --debug_objclass=30
[20:25] <joshd> otherwise the ceph command will treat it as its own option
[20:25] <rekby> i copying log part to pastebin now
[20:30] * ivotron (~ivo@adsl-76-254-17-170.dsl.pltn13.sbcglobal.net) has joined #ceph
[20:30] <rekby> http://pastebin.com/uwKKUu6e
[20:31] <rekby> i try to resize between 22:20:45 and 22:20:48
[20:32] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[20:32] <dmick> _load_class rbd from /usr/lib64/rados-classes/libcls_rbd.so
[20:32] <dmick> class rbd open got (2) No such file or directory
[20:32] <dmick> so do you have that directory, but not that file??
[20:34] <rekby> Yes, I have /usr/lib64/rados-classes/, but have no libcls_rbd.so, i have: libcls_rbd.so.1 libcls_rbd.so.1.0.0
[20:34] <rekby> ls /usr/lib64/rados-classes/
[20:34] <rekby> libcls_kvs.so libcls_kvs.so.1.0.0 libcls_lock.so.1 libcls_rbd.so.1 libcls_refcount.so libcls_refcount.so.1.0.0 libcls_rgw.so.1
[20:34] <rekby> libcls_kvs.so.1 libcls_lock.so libcls_lock.so.1.0.0 libcls_rbd.so.1.0.0 libcls_refcount.so.1 libcls_rgw.so libcls_rgw.so.1.0.0
[20:34] * gentleben (~sseveranc@208.74.182.4.static.etheric.net) Quit (Quit: gentleben)
[20:36] <rekby> I try:
[20:36] <rekby> [root@localhost ~]# cd /usr/lib64/rados-classes/
[20:36] <rekby> [root@localhost rados-classes]# ln -s libcls_rbd.so.1 libcls_rbd.so
[20:36] <rekby> and it help me:
[20:36] <rekby> rbd resize --size=20 test
[20:36] <rekby> Resizing image: 100% complete...done.
[20:37] <dmick> yes
[20:37] * dspano (~dspano@rrcs-24-103-221-202.nys.biz.rr.com) has joined #ceph
[20:38] * BillK (~BillK@58-7-146-50.dyn.iinet.net.au) has joined #ceph
[20:38] <dmick> something's damaged about that package; more aftger lunch
[20:38] <rekby> in result:
[20:38] <rekby> rpm package for centos haven't link from libcls_rbd.so to libcls_rbd.so.1.0.0
[20:39] <rekby> thanks for you
[20:39] <rekby> i have night now, good bye
[20:40] * loicd (~loic@magenta.dachary.org) has joined #ceph
[20:43] * jbarth (~jbarth@a.clients.kiwiirc.com) Quit (Quit: http://www.kiwiirc.com/ - A hand crafted IRC client)
[20:48] * BillK (~BillK@58-7-146-50.dyn.iinet.net.au) Quit (Quit: Leaving)
[20:48] * BillK (~BillK@58-7-146-50.dyn.iinet.net.au) has joined #ceph
[20:48] * fghaas (~florian@91-119-65-118.dynamic.xdsl-line.inode.at) has joined #ceph
[20:49] * fghaas (~florian@91-119-65-118.dynamic.xdsl-line.inode.at) has left #ceph
[20:50] * yasu` (~yasu`@dhcp-59-149.cse.ucsc.edu) has joined #ceph
[20:51] * gentleben (~sseveranc@c-98-207-40-73.hsd1.ca.comcast.net) has joined #ceph
[20:59] * rekby (~Adium@2.92.10.160) Quit (Quit: Leaving.)
[21:06] * rekby (~Adium@2.92.10.160) has joined #ceph
[21:06] * rekby (~Adium@2.92.10.160) Quit ()
[21:09] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) has joined #ceph
[21:11] <alexxy[home]> sage: any ideas why mds can crash immidiently on start after upgrading from 0.58 to 0.60?
[21:12] <alexxy[home]> sage: mds log http://bpaste.net/show/88751/
[21:16] * jjgalvez (~jjgalvez@12.248.40.138) has joined #ceph
[21:23] <dspano> I looked briefly in the docs. Is there a command to convert an rbd to format 2 in bobtail, or do you have to export then import your rbd to achieve this as mentioned in this post from Josh Durgin? https://bugs.launchpad.net/cinder/+bug/1115912
[21:26] * tryggvil_ (~tryggvil@rtr1.tolvusky.sip.is) has joined #ceph
[21:26] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) Quit (Read error: Connection reset by peer)
[21:26] * tryggvil_ is now known as tryggvil
[21:29] * BillK (~BillK@58-7-146-50.dyn.iinet.net.au) Quit (Read error: Connection reset by peer)
[21:33] * yanzheng1 (~zhyan@101.84.120.192) has joined #ceph
[21:39] <dspano> Nevermind. I think I found my answer here. http://www.mail-archive.com/ceph-devel@vger.kernel.org/msg10012.html
[21:40] * madkiss (~madkiss@tmo-107-76.customers.d1-online.com) has joined #ceph
[21:57] * yanzheng1 (~zhyan@101.84.120.192) Quit (Ping timeout: 480 seconds)
[22:00] * jackhill (jackhill@pilot.trilug.org) Quit (Remote host closed the connection)
[22:05] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) Quit (Quit: tryggvil)
[22:10] * fghaas (~florian@91-119-65-118.dynamic.xdsl-line.inode.at) has joined #ceph
[22:10] * fghaas (~florian@91-119-65-118.dynamic.xdsl-line.inode.at) has left #ceph
[22:12] <dmick> dspano: no translation utility, no
[22:12] * sjustlaptop (~sam@71-83-191-116.dhcp.gldl.ca.charter.com) Quit (Ping timeout: 480 seconds)
[22:16] * yanzheng1 (~zhyan@101.84.253.196) has joined #ceph
[22:18] * sjustlaptop (~sam@206.29.182.186) has joined #ceph
[22:19] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[22:19] * loicd (~loic@magenta.dachary.org) has joined #ceph
[22:24] * iggy (~iggy@theiggy.com) Quit (Remote host closed the connection)
[22:24] * iggy (~iggy@theiggy.com) has joined #ceph
[22:26] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[22:27] <BMDan> Is there a good way to mount an rbd volume locally? Specifically, I want to debug why a volume is giving me a "no boot volume available" message. qemu-nbd doesn't appear to support rbd on the backend.
[22:30] <jjgalvez> one sec, I'll check into that for you right now
[22:31] <jjgalvez> is your image already mapped to the local system? rbd showmapped
[22:31] * tryggvil (~tryggvil@17-80-126-149.ftth.simafelagid.is) has joined #ceph
[22:33] * diegows (~diegows@190.190.2.126) Quit (Ping timeout: 480 seconds)
[22:33] <BMDan> No, and it refuses to map! I figured that would be the easy solution.
[22:34] * drokita (~drokita@199.255.228.128) has joined #ceph
[22:34] <jjgalvez> any error messages while trying to map?
[22:35] <BMDan> Yeah, one sec...
[22:36] <BMDan> http://pastebin.ca/2348966
[22:36] <dspano> dmick: No worries. I'm exporting/importing as format 2 right now. Just thought I'd ask.
[22:36] <BMDan> Specifically, that's being returned by a write to /sys/bus/rbd/add
[22:36] <BMDan> [pid 8574] write(4, "10.101.0.2:6789,10.101.0.3:6789,"..., 128) = -1 ENOENT (No such file or directory)
[22:37] * yasu` (~yasu`@dhcp-59-149.cse.ucsc.edu) Quit (Remote host closed the connection)
[22:38] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[22:39] * tryggvil (~tryggvil@17-80-126-149.ftth.simafelagid.is) Quit (Quit: tryggvil)
[22:40] <drokita> Anyone know how to free up a mon after losing the other 2 partner mons. It is stuck looking for quorum and its partners are not coming back.
[22:41] <BMDan> Yeah, you mark them down, one sec, will find...
[22:41] * nhorman (~nhorman@hmsreliant.think-freely.org) Quit (Quit: Leaving)
[22:42] <drokita> Anytime I send a command to the mon, it just hangs terminally
[22:42] <BMDan> http://ceph.com/docs/master/rados/operations/add-or-rm-mons/#removing-monitors-from-an-unhealthy-cluster
[22:43] <drokita> Yeah, I read that, but step 1: ceph mon dump just locks up and never returns
[22:44] <jjgalvez> BMDan: your command looks correct according to my notes, "rbd map volume-940cb18e-dffd-4c07-827b-19cac64e5462 --pool volumes --name client.admin --secret /path/to/key" is the way I would write it out but I don't think the order of arguments is the issue here
[22:45] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) Quit (Quit: Leaving.)
[22:46] <BMDan> drokita: You don't need it; you just want to be on the host in question.
[22:46] <BMDan> drokita: That is, you can skip that step entirely.
[22:47] <drokita> oh, I see what you are saying.
[22:47] <drokita> You mean read the next line :P
[22:47] <dmick> BMDan: jjgalvez: as we discovered the other day, that's because it's format 2, right?
[22:48] <dmick> and, BMDan, wasn't it you in the conversation about libguestfs that ended with qemu's support for 9p?
[22:48] <dmick> but that was the other direction. Anyway, libguestfs should be able to help there I think
[22:48] <BMDan> dmick: Just tried bd map volume-940cb18e-dffd-4c07-827b-19cac64e5462 --pool volumes --name client.admin --secret ~/client.admin.key
[22:48] <BMDan> er, with an "r" at the start. Same result.
[22:49] <BMDan> Just figured I'd give it a go with your OoO. :)
[22:49] <BMDan> dmick: Yeah, that's why the rbd map thing doesn't work. Just thought someone might have another approach.
[22:50] <BMDan> Sorry if I'm being obtuse, but how does libguestfs help?
[22:52] <drokita> BMDan: Thanks! Working!
[22:53] <BMDan> drokita: Sure thing. :)
[22:53] * sagewk (~sage@2607:f298:a:607:a54f:880a:b8a2:844c) Quit (Ping timeout: 480 seconds)
[22:53] <dmick> BMDan: Hm, maybe I should think about the problem first. It won't unless you export the image, which you could do, but presumably are trying to avoid
[22:54] <BMDan> Right, I can export it, but then I can nbd it and have done, no?
[22:54] <BMDan> But the reimport ain't gonna be sparse or CoW from the parent.
[22:54] <dmick> it would if you could build libguestfs with rbd support
[22:54] <dmick> which might be possible
[22:54] <BMDan> I mean, meh in the big scheme of things, but not ideal.
[22:54] <dmick> but I don't know if anyone's done it
[22:54] <BMDan> What does libguestfs underly?
[22:54] <BMDan> underlie*
[22:55] <dmick> it sits on top of qemu code
[22:55] <dmick> IIRC
[22:55] <dmick> http://libguestfs.org/
[22:55] <dmick> so you manipulate the image with it, or mount it with FUSE
[22:59] <jjgalvez> it looks like building libguestfs with rbd support is already being worked on: http://rwmj.wordpress.com/2013/03/12/accessing-ceph-rbd-sheepdog-etc-using-libguestfs/
[23:00] <dmick> oh right, infernix mentioned this a few weeks ago
[23:00] <infernix> yeah there is improvement already
[23:00] <infernix> more complete support
[23:00] <dmick> speak of the devil...
[23:02] * sagewk (~sage@2607:f298:a:607:c995:b982:794a:d0eb) has joined #ceph
[23:02] <jefferai> slang: just saw your email about the samba vfs driver
[23:02] <jefferai> that's pretty damn cool
[23:02] <jefferai> that could make things much easier for some things I want to do
[23:02] <slang> jefferai: cool
[23:03] <slang> jefferai: still in testing though
[23:06] <infernix> pg 3.72 is stuck inactive for 7284.714359, current state peering, last acting [12,25]; pg 3.72 is stuck unclean for 7284.714608, current state peering, last acting [12,25];pg 3.72 is peering, acting [12,25]
[23:06] <infernix> what gives?
[23:09] <dmick> are osds 12 and 25 both healthy, up, and in?
[23:11] * themgt (~themgt@24-177-232-181.dhcp.gnvl.sc.charter.com) has joined #ceph
[23:14] <dmick> infernix: ^, and is the better guestfs support available publicly yet?
[23:14] * yanzheng1 (~zhyan@101.84.253.196) Quit (Ping timeout: 480 seconds)
[23:15] <infernix> yeah they are in
[23:15] <infernix> yes
[23:15] <infernix> standby, phone :>
[23:15] <dmick> k
[23:16] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) has joined #ceph
[23:18] * BMDan (~BMDan@static-108-44-155-130.clppva.fios.verizon.net) Quit (Quit: Leaving.)
[23:18] * xmltok (~xmltok@pool101.bizrate.com) Quit (Read error: Connection reset by peer)
[23:18] <pioto> hi, are there any recommended best practices for making backups of a ceph cluster? it seems that's still "TODO" here, at least: http://ceph.com/docs/master/architecture/#client-interfaces
[23:18] * xmltok (~xmltok@pool101.bizrate.com) has joined #ceph
[23:18] * sjustlaptop (~sam@206.29.182.186) Quit (Ping timeout: 480 seconds)
[23:24] <jjgalvez> generally scale out architecture doesn't involve backups (add more osds, ceph handles replication/healing), but if you're trying to backup a whole cluster you'll basically be going the way of raid again
[23:25] <drokita> Even scale out architectures are subject to logical corruption of the data itself. Backups should still be done.
[23:25] <jjgalvez> true, and beyond that disaster recovery is planned for the D release which would be helpful in these cases
[23:26] <pioto> drokita: exactly
[23:26] <pioto> i'm talkiing about recovering from "oops, just deleted all the data"
[23:26] <pioto> or, well, "just deleted this very important $blah"
[23:27] <pioto> recovering 1000s of TBs of data is just unreasonable, and hopefully that doesn't come up
[23:27] <pioto> but more mundane stuff can
[23:27] <drokita> Today, the only way you are going to get that is to use something that is aware of the internal system. For example, doing rsync on the mounted rbd
[23:27] <pioto> ok, that was my guess
[23:28] <dmick> yeah. we are working on rbd 'diffs' as a first step toward a useful disaster recovery
[23:28] <pioto> yeah, i saw some references to that
[23:28] <mjevans> Snapshots are not backups
[23:28] <dmick> but it's very new.
[23:28] <mjevans> Backups are offline or at least completely isolated form that system
[23:28] <pioto> mjevans: no, not alone. but multiple versioned snapshots somewhere elase are
[23:28] <mjevans> Snapshots, however, can be really useful speedy recovery tools
[23:28] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) Quit (Ping timeout: 480 seconds)
[23:29] <dmick> mjevans: no, but you could back up the image, and then periodically the diff, more practically than backing up the entire image regularly. But yes, there are reasons to have better/more tools around
[23:29] <pioto> for cephfs, i assume basically just 'mount & rsync' si the plan, too
[23:29] <pioto> or maybe 'mount, mkdir .snap/backup-20130403, rsync that'
[23:30] <pioto> for objects, it seems there's a 'rados export', which the --help seems to imply only will export changed objects, etc. and then you can just tar or rsync that up?
[23:30] <pioto> and recovery could be something like "here's $YOURPOOL-recovered-20130403, get what you need"
[23:30] <dmick> not aware of rados export; there's 'put'
[23:31] <pioto> (import that backup into a new pool)
[23:31] <dmick> and there's 'rbd export'
[23:31] <dmick> both of them are whole-thing copies
[23:31] <pioto> maybe rados export is new
[23:31] <dmick> oh duh, pool export, sorry
[23:31] <dmick> yes
[23:31] <pioto> yeah
[23:31] <pioto> and yes, for specific objects, get
[23:31] <dmick> I actually don't know how pool export works
[23:32] <pioto> but that's just unweildy for backups unless you have Some Important Object
[23:32] <tchmnkyz-> i have a hudge problem
[23:32] <tchmnkyz-> my ceph cluster just seems to have lost my vm's drive image
[23:32] <tchmnkyz-> the vm will not boot says could not find bootable disk
[23:33] <dmick> pioto: it looks like "every object in the pool" to me.
[23:33] <dmick> I actually want to experiment with that.
[23:33] <tchmnkyz-> someone please help me
[23:33] <jjgalvez> tchmnkyz: what is the status of your ceph cluster?
[23:33] <pioto> tchmnkyz-: are you using the kernel rbd driver or the qemu one? I think i saw that with the kernel one, and fixed by reloading the kernel module or something. or maybe just by switching to qemu
[23:33] <tchmnkyz-> qemu
[23:33] <pioto> and it happened when i was testing rebooting the cluster
[23:33] <pioto> hm
[23:34] <tchmnkyz-> i was in the middle of migrating one node at a time from debian to ubuntu
[23:34] <pioto> well. if you get qemu to at least start, then it's finding the image
[23:34] <tchmnkyz-> it was the last node
[23:34] <pioto> and it seems to just be corrupt i guess?
[23:34] <tchmnkyz-> i need to find a way to recover thsoe two images
[23:34] <tchmnkyz-> at all costs
[23:34] <pioto> maybe try to bring up a recovery image in there and fsck or something?
[23:34] <dmick> what's rbd say about them
[23:35] <tchmnkyz-> never really interacted with rbd
[23:35] <tchmnkyz-> not directly
[23:36] <tchmnkyz-> when i do a list they are all there
[23:36] <infernix> hm
[23:36] <infernix> so
[23:36] <infernix> pg 3.72 is stuck inactive for 7284.714359, current state peering, last acting [12,25]; pg 3.72 is stuck unclean for 7284.714608, current state peering, last acting [12,25];pg 3.72 is peering, acting [12,25]
[23:36] <infernix> osd 12 is having problems, disk is reporting media errors
[23:37] <infernix> but actually it's not that disk
[23:37] <infernix> it's another disk that's broken in that one, osd 15
[23:37] <tchmnkyz-> any suggestions on how i can recover this
[23:37] <infernix> so both osd 12 and 25 are fine
[23:38] <tchmnkyz-> like i just checked one of the virtal disks
[23:38] <tchmnkyz-> it seems there is no partitions on the drive at all
[23:40] <dmick> infernix: I assume you've read the troubleshooting section on ceph.com. are 12,25 the originla mapping set for those pgs? (pg query, probably)
[23:40] <dmick> hm, actually, how do you tell the original mapping set. Anyway, if 12 is having problems, that could be a clue too
[23:40] <infernix> the pg query doesn't say much
[23:41] <infernix> 12 and 25 should really be just fine, no media errors whatsoever
[23:41] <tchmnkyz-> rbd: add failed: (2) No such file or directory
[23:41] <tchmnkyz-> it basically looks like it is missing
[23:41] <tchmnkyz-> even though ls shows it
[23:41] <dmick> sorry, I mean 'if 15 is having problems'
[23:41] <tchmnkyz-> somone please help i need to recover this
[23:41] <dmick> tchmnkyz-: not sure why you're trying rbd add, but that fails for other reasons
[23:42] <dmick> "when i do a list"..you mean rbd ls?
[23:42] <tchmnkyz-> yes
[23:42] <dmick> what about rbd info
[23:42] <dmick> on one of them
[23:42] <tchmnkyz-> k
[23:43] <tchmnkyz-> rbd: error opening image vm-5003-disk-12013-04-03 16:43:21.015400 7f691f734780 -1 librbd::ImageCtx: error finding header: (2) No such file or directory
[23:43] <tchmnkyz-> : (2) No such file or directory
[23:43] <tchmnkyz-> everything was there
[23:43] <tchmnkyz-> how does it just go poof
[23:43] <tchmnkyz-> i cant have things just going poof
[23:43] <dmick> I think everyone everywhere would agree that things going poof is a bad thing and should be avoided
[23:44] <dmick> were these format 1 or format 2 images?
[23:44] <infernix> so the pg is stuck inactive and unclean, is in state peering - i have no unfound objects
[23:44] <drokita> Anyone familiar with this one: 2013-04-03 16:44:10.197094 7f78a379a700 monclient: hunting for new mon
[23:44] <tchmnkyz-> so dmick any suggestions how this could be corrected?
[23:44] <dmick> (02:44:08 PM) dmick: were these format 1 or format 2 images?
[23:44] <infernix> no down_osds_we_would_probe, no peering_blocked_by
[23:45] <tchmnkyz-> sorry missed it
[23:45] <tchmnkyz-> format 2
[23:45] <jjgalvez> drokita: monitor processes still running?
[23:45] <dmick> jjgalvez: maybe you can help infernix
[23:45] <drokita> It is, and the other servers can see it
[23:46] <tchmnkyz-> i just noticed that there was a error in my info line
[23:46] <drokita> just this server is scrolling those errors when I try to get a ceph status
[23:46] <tchmnkyz-> i forgot the pool
[23:46] <tchmnkyz-> root@san03:~ # rbd info vm-5003-disk-1 -p moto
[23:46] <tchmnkyz-> rbd image 'vm-5003-disk-1': size 65536 MB in 16384 objects order 22 (4096 KB objects) block_name_prefix: rbd_data.a66d482ae8944a format: 2 features: layering, striping stripe unit: 4096 KB stripe count: 1
[23:46] <tchmnkyz-> so it has data
[23:46] <dmick> ok, good, so it's not gone
[23:46] <tchmnkyz-> how can i regain access to it?
[23:47] <pioto> tchmnkyz-: can you export it, and see if the export looks reasonable?
[23:47] <pioto> like, do an `fsck -l` on it or whatever
[23:47] <jjgalvez> infernix: I've seen pgs in a stuck peering state before, with the osds up an in, not sure what causes it but restarting each of the osds (one at a time) has cleared it up for me before
[23:47] <pioto> err, fdisk -l, and an fsck on the actual partitions
[23:47] <dmick> tchmnkyz-: what pioto said is what I'd try now, probably
[23:48] <jjgalvez> infernix: restart one, wait for recovery, check status, restart the other if not cleared
[23:48] <dmick> i.e. treat it as "the VM damaged the disk data so that the disk is unbootable" until proven otherwise
[23:48] <infernix> jjgalvez: 10-4; i will wait until i have the broken OSD recovered first
[23:49] <pioto> i recently noticed that rbd has ways to lock an image... is there a way to tell libvirt/qemu to lock that image?
[23:49] <pioto> which i assume would prevent me from doing 'dumb stuff' to it?
[23:49] <pioto> or, is that not what i think it is?
[23:49] <dmick> pioto: it's only cooperative; everyone must check for locks and behave correctly
[23:49] <dmick> I don't know if qemu can do it
[23:50] <pioto> ok, so like normal flock() and such. got it.
[23:50] <pioto> in terms of being 'advisory';
[23:50] <pioto> hm
[23:51] <tchmnkyz-> i am trying to export it now
[23:51] <tchmnkyz-> will see how it goes
[23:51] <tchmnkyz-> the other vm it was only the second disk that was bad
[23:51] <tchmnkyz-> it has no partitions or nothing
[23:51] <tchmnkyz-> no lvms
[23:51] <dmick> you could also try mounting it as a nonboot drive and examining it from the vm
[23:51] <tchmnkyz-> like it is a brand new disk
[23:52] <pioto> or, poke it with libguestfs and the like
[23:52] <tchmnkyz-> ok that one is exported
[23:52] <tchmnkyz-> not sure where to take it from here
[23:52] <tchmnkyz-> never really had this problem before
[23:53] <pioto> well. start with an `fdisk -l`
[23:53] <pioto> see if it has a partition table in it
[23:55] <tchmnkyz-> Error: /root/vm-5003-disk-1.raw: unrecognised disk label is what i am getting
[23:56] <tchmnkyz-> fdisk and parted cant find a partition table
[23:56] <tchmnkyz-> i could not even begin to remember the partition table for this stuff
[23:58] <tchmnkyz-> what i dont get is my cluster is full up and in full health
[23:58] <tchmnkyz-> please someone help me fix this

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.