#ceph IRC Log


IRC Log for 2012-08-13

Timestamps are in GMT/BST.

[0:05] * s[X] (~sX]@ppp59-167-154-113.static.internode.on.net) Quit (Remote host closed the connection)
[0:06] * s[X] (~sX]@ppp59-167-154-113.static.internode.on.net) has joined #ceph
[0:08] * s[X] (~sX]@ppp59-167-154-113.static.internode.on.net) Quit (Remote host closed the connection)
[0:17] * lxo (~aoliva@lxo.user.oftc.net) Quit (Quit: later)
[0:31] * MarkN (~nathan@ has joined #ceph
[0:38] * MarkN (~nathan@ has left #ceph
[1:11] <joshd> iggy: that plugin works without the cephfs kernel client, since it uses libcephfs, if you can use the kernel client then re-exporting with samba should work I think
[1:12] <joshd> iirc the reason for its development was to avoid running a kernel client on the same host as an osd
[1:24] * Cube (~Adium@cpe-76-95-223-199.socal.res.rr.com) Quit (Quit: Leaving.)
[1:28] * tnt (~tnt@17.127-67-87.adsl-dyn.isp.belgacom.be) Quit (Ping timeout: 480 seconds)
[1:41] * Cube (~Adium@cpe-76-95-223-199.socal.res.rr.com) has joined #ceph
[2:05] * loicd1 (~loic@brln-4dbab807.pool.mediaWays.net) Quit (Ping timeout: 480 seconds)
[2:06] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[2:10] * Cube1 (~Adium@cpe-76-95-223-199.socal.res.rr.com) has joined #ceph
[2:16] * Cube (~Adium@cpe-76-95-223-199.socal.res.rr.com) Quit (Ping timeout: 480 seconds)
[2:22] * steki-BLAH (~steki@ Quit (Quit: Ja odoh a vi sta 'ocete...)
[2:22] * kingleecher (~kingleech@204-174-98-35.dhcp470.dsl.ucc-net.ca) Quit (Quit: Leaving)
[2:30] * mpw2 (~mpw@chippewa-nat.cray.com) Quit (Quit: Nettalk6 - www.ntalk.de)
[2:43] * bchrisman (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[2:44] * bchrisman (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) has joined #ceph
[3:32] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[3:42] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[3:45] * yoshi (~yoshi@p22043-ipngn1701marunouchi.tokyo.ocn.ne.jp) has joined #ceph
[3:53] * Cube (~Adium@cpe-76-95-223-199.socal.res.rr.com) has joined #ceph
[3:58] * Cube1 (~Adium@cpe-76-95-223-199.socal.res.rr.com) Quit (Ping timeout: 480 seconds)
[3:58] * sagelap (~sage@cpe-76-94-40-34.socal.res.rr.com) Quit (Read error: Connection timed out)
[4:02] * sagelap (~sage@cpe-76-94-40-34.socal.res.rr.com) has joined #ceph
[4:10] * Qten (qgrasso@qten.qnet.net.au) has joined #ceph
[4:12] * Qten (qgrasso@qten.qnet.net.au) Quit ()
[5:00] * bchrisman1 (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) has joined #ceph
[5:05] * bchrisman (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[5:32] * Qten (Q@qten.qnet.net.au) has joined #ceph
[5:35] * renzhi (~renzhi@raq2064.uk2.net) has joined #ceph
[6:00] <iggy> makes sense
[6:12] * EmilienM (~EmilienM@ has joined #ceph
[6:14] * s[X] (~sX]@ppp59-167-154-113.static.internode.on.net) has joined #ceph
[6:33] * renzhi (~renzhi@raq2064.uk2.net) Quit (Ping timeout: 480 seconds)
[6:42] <Tobarja> joshd: thanks. now, if i could just find a full example of how to put it into smb.conf
[6:47] * renzhi (~renzhi@ has joined #ceph
[7:06] * renzhi (~renzhi@ Quit (Quit: Leaving)
[7:23] * deepsa_ (~deepsa@ has joined #ceph
[7:24] * deepsa (~deepsa@ Quit (Ping timeout: 480 seconds)
[7:24] * deepsa_ is now known as deepsa
[7:58] * tnt (~tnt@17.127-67-87.adsl-dyn.isp.belgacom.be) has joined #ceph
[8:25] <exec> tnt: hi. how is the status of your evaluating? have you got stable cluster?
[8:27] <tnt> exec: well I got a kernel panic over the week end, so I'd say "no" :p
[8:32] <exec> tnt: heh. I've broken cluster after the weekend too. looking into details :(
[8:33] <exec> hope to restore data on it. it's just a copy of backups, but I'd like to be sure it is possible )
[9:08] * s[X] (~sX]@ppp59-167-154-113.static.internode.on.net) Quit (Remote host closed the connection)
[9:08] * s[X] (~sX]@ppp59-167-154-113.static.internode.on.net) has joined #ceph
[9:09] <exec> journal size increase helped. (increase/flush/rm/mkjournal)
[9:13] * tnt (~tnt@17.127-67-87.adsl-dyn.isp.belgacom.be) Quit (Ping timeout: 480 seconds)
[9:13] * s[X] (~sX]@ppp59-167-154-113.static.internode.on.net) Quit (Quit: Leaving...)
[9:14] * verwilst (~verwilst@d5152FEFB.static.telenet.be) has joined #ceph
[9:18] <exec> any real suggestions about "max journal size/osd max write size" parameters?
[9:28] * BManojlovic (~steki@ has joined #ceph
[9:33] * loicd (~loic@brln-4dbab807.pool.mediaWays.net) has joined #ceph
[9:59] * Leseb (~Leseb@ has joined #ceph
[10:09] * joshd (~jdurgin@2602:306:c5db:310:1e6f:65ff:feaa:beb7) Quit (Ping timeout: 480 seconds)
[10:23] * fc (~fc@ has joined #ceph
[10:31] <loicd> Leseb: what kind of feedback are you looking for regarding http://ceph.com/wiki/Benchmark ?
[10:38] * tnt (~tnt@212-166-48-236.win.be) has joined #ceph
[10:40] <Leseb> loicd: simple feedback like first assumption and feeling after the benchmarks, disappointment etc etc??? general view
[10:41] <loicd> Leseb: it would be good to have more benchmark results published in a consistent way. At the moment there is almost nothing.
[10:45] * pmjdebruijn (~pmjdebrui@overlord.pcode.nl) has joined #ceph
[10:45] <pmjdebruijn> hi folks
[10:45] <Leseb> loicd: this statement is about to change this week
[10:46] <pmjdebruijn> does anybody know if there is any ballpark release schedule for 0.48.1 yet?
[10:48] * kermin (ca037809@ircip3.mibbit.com) has joined #ceph
[10:48] <kermin> hi all !
[10:49] <kermin> hello ppl ?
[10:51] <Deuns> hey
[10:54] <kermin> hi Deuns
[11:17] * yoshi (~yoshi@p22043-ipngn1701marunouchi.tokyo.ocn.ne.jp) Quit (Remote host closed the connection)
[11:45] <NaioN> anybody with recovery knowledge awake?
[11:46] <NaioN> I have a pg that's stuck in recovering with 1 unfound object
[11:47] <NaioN> with ceph pg 4.700 query (the pg=4.700) it tries to query a small list of osds in search for a possible unfound object
[11:48] <NaioN> all osds reply exept 1 so it stays in quering that osd and I can't use the ceph pg 4.700 mark_unfound_lost revert
[11:48] <NaioN> because it says it has one resource yet to query for the unfound object
[11:49] <tnt> is there a reason that osd doesn't reply ?
[11:55] <Deuns> if i'm not mistaken "osdmaptool --test-map-pg 1.00 /tmp/osdmap" should tell me on which osd is a pg, right ?
[12:02] <NaioN> tnt: can't find anything in the logs
[12:08] <tnt> NaioN: so the osd is up and nothin happened to it ? (ie no crash or anything)
[12:09] <NaioN> indeed
[12:11] <NaioN> I only see this: 2012-08-13 10:23:20.521479 7f5061bae700 0 osd.65 45442 pg[4.700( v 31255'4 lc 23489'10296 (0'0,31255'4] n=0 ec=79 les/c 45442/28743 45440/45440/45440) [65,44] r=0 lpr=45440 pi=28705-45439/125 rops=1 lcod 0'0 mlcod 0'0 active+recovering m=
[12:11] <NaioN> 1] _failed_push e4152700/edugrip-vdr01.rbd/head//4 from osd.58, reps on 58
[12:11] <NaioN> but it is stuck quering on osd 54 not 58
[12:11] <NaioN> so 58 clearly doesn't have the object it is searching for
[12:12] <NaioN> but osd 54 doesn't even have a dir 4.700_head under current
[12:13] <NaioN> tnt: I see this in the log of 54: 2012-08-13 11:53:55.502291 7f88979d2700 0 -- >> pipe(0x6cde780 sd=147 pgs=754 cs=1 l=0).fault with nothing to send, going to standby
[12:13] <NaioN> about 6 times
[12:13] <NaioN> but I don't know if it's related to the query of osd 65
[12:32] * nhorman (~nhorman@2001:470:8:a08:7aac:c0ff:fec2:933b) has joined #ceph
[13:29] * deepsa (~deepsa@ Quit (Quit: ["Textual IRC Client: www.textualapp.com"])
[13:29] * deepsa (~deepsa@ has joined #ceph
[13:37] * shdb (~shdb@80-219-123-230.dclient.hispeed.ch) Quit (Ping timeout: 480 seconds)
[13:41] * deepsa_ (~deepsa@ has joined #ceph
[13:42] * deepsa (~deepsa@ Quit (Ping timeout: 480 seconds)
[13:42] * deepsa_ is now known as deepsa
[14:20] * kermin (ca037809@ircip3.mibbit.com) Quit (Quit: http://www.mibbit.com ajax IRC Client)
[14:22] * The_Bishop (~bishop@2a01:198:2ee:0:58dd:eb9d:5531:9ea8) has joined #ceph
[14:23] <tnt> mmm, RBD volume seem to be always corrupted no matter what I do.
[14:30] <Leseb> hi guys
[14:39] * deepsa_ (~deepsa@ has joined #ceph
[14:43] * deepsa (~deepsa@ Quit (Ping timeout: 480 seconds)
[14:43] * deepsa_ is now known as deepsa
[14:48] * lightspeed (~lightspee@ Quit (Ping timeout: 480 seconds)
[14:48] <tnt> libceph: corrupt inc osdmap epoch 342 off 98 (ffffc90011a1707e of ffffc90011a1701c-ffffc90011a1707e)
[14:48] <tnt> this sounds bad
[14:51] <tnt> sounds like http://tracker.newdream.net/issues/2446
[14:52] <tnt> So I need a new kernel for my dom0 ...
[14:54] <darkfader> tnt: i wish there was some genius who writes a stub xen domain for ceph storage backend and documents that
[14:54] <darkfader> of course, flying to mars from your next bus station is more likely to happen
[14:55] <tnt> stub xen domain ? You mean just the OSD in a domU ?
[14:58] <tnt> personally I would like xen not to use the kernel driver but use the userspace librdb, even for PV domains.
[15:02] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[15:04] <darkfader> actually, a special domU that just runs the client / backend driver. and the domUs then get their storage not from dom0 but that domU
[15:04] <darkfader> that's the "fast" way of doing things but almost never used in practice because it's so complex
[15:05] <darkfader> like, uh, fujitsu uses it, they have xen in their smaller mainframes
[15:05] <darkfader> and there you need reliable/qos-able storage
[15:05] <darkfader> uh anyway
[15:05] <darkfader> it will never happen :)\
[15:10] <Deuns> is there a way to know if a my pool is replicated ?
[15:10] <Deuns> I tried "ceph osd pool set rbd size 3"
[15:11] <tnt> thn it will be replicated 3 times.
[15:11] <tnt> you can monitor the status with 'ceph -w'
[15:13] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[15:14] <Deuns> tnt: when I do change the size, I don't see anything changing with ceph -w
[15:23] <NaioN> tnt: if i'm correct I had the same issue, but with a kernel upgrade it went away
[15:23] * hijacker (~hijacker@ Quit (Read error: Connection reset by peer)
[15:24] <tnt> NaioN: yes I think so too. Unfortunately that means I have to somehow remember how to create a debian package for kernel ...
[15:24] <NaioN> hehe
[15:24] <NaioN> I have pmjdebruijn for that! :)
[15:24] * hijacker (~hijacker@ has joined #ceph
[15:25] <NaioN> but we use Ubuntu, don't know if those also work for you, but it's on a ppa
[15:25] * pmjdebruijn giggles
[15:25] <tnt> I've also been pointed to blktap which would allow to use the userspace rbd library for DomU block backend rather than the kernel driver.
[15:25] <pmjdebruijn> tnt: make-kpkg works best if you need something quick and dirty
[15:25] * dspano (~dspano@rrcs-24-103-221-202.nys.biz.rr.com) has joined #ceph
[15:25] <NaioN> tnt: user-space would be very nice
[15:26] <NaioN> then you can use the newer librbd's
[15:26] <pmjdebruijn> bypassing kernel i/o involvement is generally a good thing anyhow with regard to stability
[15:26] <tnt> yes exactly :P
[15:27] <pmjdebruijn> no deadlocked processes with failures
[15:27] <pmjdebruijn> and the likes
[15:27] <NaioN> and you get more features! :)
[15:27] <tnt> more stable, more up-to-date and supports read/write caching ...
[15:27] <NaioN> indeed
[15:28] <darkfader> hehe. two people praysing non-sync writes ;p
[15:28] <darkfader> all you're getting is dataloss
[15:28] <NaioN> the trouble for us is we use the rbd's for re-exporting
[15:28] <pmjdebruijn> darkfader: bypassing the kernel doesn't mean async per-se does it?
[15:28] <darkfader> but isn't the problem not the locking but the non-resuming?
[15:28] <tnt> darkfader: if you respect the 'sync' calls that should not be an issue.
[15:29] <darkfader> but then it would block
[15:29] <NaioN> darkfader: yes it blocks
[15:29] <tnt> yes, but I want it to block whenever the FS requests it, but I don't want it do block when the upper layer doesn't need it to sync.
[15:29] <NaioN> but not on a kernel io call
[15:29] <darkfader> ah ok
[15:29] <darkfader> thanks :)
[15:30] <tnt> Currently it syncs for _all_ write ... I only want it to sync whenever the upper layer needs it to.
[15:30] <pmjdebruijn> the basic idea is that a single VM with an I/O problems can't affect your kernel
[15:31] <NaioN> and it's easier to kill the vm/process because it doesn't lock on a kernel IO call
[15:31] <darkfader> i used to be in a finance shop, there'd you'd want the server to block if one byte can't be written, because the rest would be non-trusted. but of course if I think of different VMs that is completely useless. they can sort it out in their own kernel
[15:31] <darkfader> glad I asked
[15:31] <darkfader> and thanks again :)
[15:32] <pmjdebruijn> darkfader: in such a use-case one could ask if "complicated storage" should be used at all :)
[15:32] <darkfader> hehe :>
[15:32] <pmjdebruijn> and not per-se just ceph
[15:32] <pmjdebruijn> you get the jist of it
[15:32] <NaioN> yeah also for recovery
[15:33] <darkfader> it was happy times, just a few big emc's and no mid-range ones. so you'd have 512GB+ of cache instead of scaling out
[15:33] <NaioN> for our case it would be nice to have a LIO to Ceph module
[15:33] <darkfader> but they switched to icky little netapps
[15:33] <pmjdebruijn> NaioN: librbd module :)
[15:33] <NaioN> indeed
[15:33] <darkfader> NaioN: i know someone who works for the LIO company and he loves ceph
[15:33] <pmjdebruijn> well
[15:33] <NaioN> please ask him for a module!!!! :)
[15:34] <pmjdebruijn> it's a pretty nice match
[15:34] * The_Bishop (~bishop@2a01:198:2ee:0:58dd:eb9d:5531:9ea8) Quit (Ping timeout: 480 seconds)
[15:34] <darkfader> NaioN: write to the target-devel list, the main dev is extremely nice and smart
[15:34] <darkfader> (or whatever the list is called)
[15:35] <NaioN> hmmmm will have a look
[15:35] <darkfader> let me knwo if you can't find it
[15:35] <darkfader> but it's listed somewhere on linux-iscsi.org
[15:35] <NaioN> k thx
[15:37] <NaioN> darkfader: found it http://vger.kernel.org/vger-lists.html#target-devel
[15:40] * The_Bishop (~bishop@2a01:198:2ee:0:346b:e850:d17:ac41) has joined #ceph
[16:10] <tnt> Is anyone using Xen here ? When I setup the rbd device in the dom0 and then use that as the backing device for a domU, all hell breaks loose and I get corruption directly.
[16:16] * EmilienM (~EmilienM@ Quit (Read error: Connection reset by peer)
[16:23] * EmilienM (~EmilienM@ has joined #ceph
[16:56] * CristianDM (~CristianD@ has joined #ceph
[16:56] <CristianDM> Hi.
[16:57] <CristianDM> I have issues with ceph -s
[16:58] <CristianDM> Ceph reply unrecognized subsystem
[16:58] <CristianDM> With ceph version 0.48argonaut (commit:c2b20ca74249892c8e5e40c12aa14446a2bf2030)
[16:58] <CristianDM> Any idea how to fix it?
[17:09] * The_Bishop (~bishop@2a01:198:2ee:0:346b:e850:d17:ac41) Quit (Ping timeout: 480 seconds)
[17:14] * BManojlovic (~steki@ Quit (Quit: Ja odoh a vi sta 'ocete...)
[17:17] * The_Bishop (~bishop@2a01:198:2ee:0:58dd:eb9d:5531:9ea8) has joined #ceph
[17:20] * verwilst (~verwilst@d5152FEFB.static.telenet.be) Quit (Quit: Ex-Chat)
[17:24] * glowell (~Adium@c-98-210-226-131.hsd1.ca.comcast.net) has joined #ceph
[17:24] * joshd (~jdurgin@2602:306:c5db:310:1e6f:65ff:feaa:beb7) has joined #ceph
[17:26] * loicd1 (~loic@brln-4dbab6b9.pool.mediaWays.net) has joined #ceph
[17:30] * loicd (~loic@brln-4dbab807.pool.mediaWays.net) Quit (Read error: Operation timed out)
[17:31] * mpw (~mpw@chippewa-nat.cray.com) has joined #ceph
[17:39] * bchrisman1 (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[17:43] * Leseb (~Leseb@ Quit (Ping timeout: 480 seconds)
[17:45] * sage (~sage@cpe-76-94-40-34.socal.res.rr.com) Quit (Read error: Operation timed out)
[17:48] * tnt (~tnt@212-166-48-236.win.be) Quit (Ping timeout: 480 seconds)
[17:52] <gregaf> CristianDM: the only time I've seen that error (unrecognized subsystem) with ceph status is with a v0.48 ceph tool and an older monitor
[17:53] <CristianDM> But if I have the last updates of ceph
[17:53] <CristianDM> How I can have an old monitor?
[17:55] <gregaf> did you restart the monitor?
[17:56] <CristianDM> Yes I restart all
[17:57] <CristianDM> service ceph restart
[17:57] <CristianDM> This restart all
[17:57] <gregaf> only on the box it's running on
[17:57] <CristianDM> service ceph -a restart
[17:57] <gregaf> that *should* do it, yes
[17:58] <gregaf> but honestly it's a little fragile
[17:58] <gregaf> I'd ssh in to each monitor and do a restart there, and if you still have the problem we can check it out in more detail (get logs going, etc)
[17:59] <CristianDM> I will to reboot the servers
[18:00] * The_Bishop (~bishop@2a01:198:2ee:0:58dd:eb9d:5531:9ea8) Quit (Ping timeout: 480 seconds)
[18:00] <CristianDM> Another doubt. Any SSD to recomend for storage journaling?
[18:00] <CristianDM> Ocz Vertex or Samsung are good?
[18:01] * BManojlovic (~steki@ has joined #ceph
[18:02] * sage (~sage@cpe-76-94-40-34.socal.res.rr.com) has joined #ceph
[18:02] <gregaf> as far as I know most reputable SSD vendors these days are doing fine
[18:02] <CristianDM> Perfect. This is a sequential write (journal) really?
[18:03] <gregaf> yep!
[18:04] <CristianDM> Perfect. I will need to put SSD. I have low performance with small files
[18:09] * The_Bishop (~bishop@2a01:198:2ee:0:346b:e850:d17:ac41) has joined #ceph
[18:09] <CristianDM> Wops
[18:10] <CristianDM> I reboot server and
[18:10] <CristianDM> unable to read magic from mon data.. did you run mkcephfs?
[18:14] * joshd (~jdurgin@2602:306:c5db:310:1e6f:65ff:feaa:beb7) Quit (Quit: Leaving.)
[18:17] * Tv_ (~tv@2607:f298:a:607:60e1:881b:ec67:50fe) has joined #ceph
[18:23] <gregaf> CristianDM: that's not good ?????do you have any logging?
[18:26] <gregaf> NaioN: is your PG still stuck recovering?
[18:32] * Cube (~Adium@cpe-76-95-223-199.socal.res.rr.com) Quit (Quit: Leaving.)
[18:36] * tnt (~tnt@17.127-67-87.adsl-dyn.isp.belgacom.be) has joined #ceph
[18:36] * bchrisman (~Adium@ has joined #ceph
[18:37] * joao (~JL@ has joined #ceph
[18:41] * Deuns (~kvirc@ Quit (Quit: KVIrc 4.0.0 Insomnia http://www.kvirc.net/)
[19:07] <CristianDM> gregaf: The issue are with the mounts
[19:07] <CristianDM> But now I upgrade to 0.49
[19:07] <CristianDM> And have the same issue
[19:09] * Ryan_Lane (~Adium@c-67-160-217-184.hsd1.ca.comcast.net) has joined #ceph
[19:10] * joshd (~joshd@2607:f298:a:607:221:70ff:fe33:3fe3) has joined #ceph
[19:12] * chutzpah (~chutz@ has joined #ceph
[19:12] <CristianDM> 2012-08-13 14:11:25.985279 7fdda00d9700 1 mon.b@0(leader).osd e89 e89: 6 osds: 6 up, 6 in
[19:12] <CristianDM> 2012-08-13 14:11:29.817963 7fdda00d9700 1 mon.b@0(leader).osd e90 e90: 6 osds: 6 up, 6 in
[19:13] * Cube (~Adium@ has joined #ceph
[19:14] <gregaf> CristianDM: okay, so not all your monitors are failing to start up?
[19:15] * dmick (~dmick@ has joined #ceph
[19:15] <CristianDM> The monitors start, but I can??t get info with "ceph -s"
[19:15] <gregaf> on the monitor that is failing, could you go look at your monitor's data directory and see if it has a file "magic"?
[19:15] <CristianDM> The monitors are all up
[19:15] <CristianDM> But when I run "ceph -s" have unrecognized subsystem
[19:15] <gregaf> are you sure?
[19:16] <gregaf> I thought you said that they were printing an error message "unable to read magic from mon data.. did you run mkcephfs"
[19:16] <gregaf> the next thing they do after that is quit
[19:16] <CristianDM> Yes, but i reply that is an issue with the mount point
[19:16] <gregaf> ah, got it
[19:17] <CristianDM> After reboot
[19:17] <gregaf> so now they're starting?
[19:17] <gregaf> can you pastebin the beginning of their log?
[19:17] <CristianDM> I mount and the monitors starts
[19:17] <CristianDM> yes, wait
[19:17] <gregaf> brb
[19:18] <CristianDM> http://pastie.org/4467228
[19:19] * maelfius (~Adium@ has joined #ceph
[19:19] * Ryan_Lane (~Adium@c-67-160-217-184.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[19:20] * Ryan_Lane (~Adium@c-67-160-217-184.hsd1.ca.comcast.net) has joined #ceph
[19:30] <NaioN> gregaf: yeps
[19:42] <gregaf> NaioN: okay, can you give me the output of ceph pg dump for that PG? (ceph pg dump | grep "pgid")
[19:43] <NaioN> gregaf: mom
[19:43] <NaioN> 4.700 6 1 8 1 23429120 24292 24292 active+recovering 2012-08-13 11:36:07.856073 45666'180 45540'110132 [65,44] [65,44] 13565'10291 2012-08-08 00:03:59.175571
[19:43] * eightyeight (~atoponce@pinyin.ae7.st) Quit (Remote host closed the connection)
[19:45] <NaioN> I used ceph pg 4.700 query
[19:50] <gregaf> CristianDM: you said you're using argonaut, correct?
[19:50] <gregaf> can you run "ceph ???version" for me?
[19:51] <NaioN> gregaf: the problem is that pg 4.700 thinks it maybe has a unfound object on osd 54
[19:52] <NaioN> but osd 54 doesn't have a directory 4.700_head so it clearly doesn't has the object
[19:53] <NaioN> but with ceph pg 4.700 query I see that it only doesn't get a response from osd 54
[19:53] <gregaf> yeah, sorry, I was trying to get Sam in to look at this with you since he's more familiar with that code than I am
[19:53] <NaioN> oh ok
[19:54] <NaioN> well at the moment we made a new rbd, but I doubt if i can remove the old one
[19:54] <NaioN> because even quering for info on the rbd fails
[19:54] <gregaf> okay
[19:54] <gregaf> I think he should be available in a few minutes
[19:55] <gregaf> pmjdebruijn: sounds like .48.1 should be out today, maybe tomorrow if Sage gets too busy
[19:55] <gregaf> :)
[19:55] <NaioN> nice!
[19:55] <NaioN> (for both comments :))
[19:56] <NaioN> I can have pmjdebruijn build new images for the servers tomorrow :)
[20:04] <sjust> NaioN: catching up, one sec
[20:08] <sjust> NaioN: can you post the results of the query?
[20:09] <NaioN> k
[20:09] <NaioN> mom
[20:10] <NaioN> sjust: http://pastebin.com/scUqGGqF
[20:11] <NaioN> 54 is not queried because i restarted it
[20:11] <sjust> is it up yet?
[20:11] <sjust> oh...
[20:11] <NaioN> but if i restart 65 it will stuck in quering
[20:12] * asadpanda (~asadpanda@2001:470:c09d:0:20c:29ff:fe4e:a66) Quit (Ping timeout: 480 seconds)
[20:12] <NaioN> i can restart 65
[20:12] <sjust> yeah, do that
[20:12] <sjust> wait
[20:12] <sjust> can you turn on osd debugging on 65 and 54?
[20:12] <NaioN> but 54 doesn't even have a 4.700_head directory while all other osds have one (that are listed)
[20:13] <NaioN> yeah sure
[20:13] <NaioN> which value?
[20:13] <sjust> yeah, that's the real question, that means that 54 should not show up in the might_have_unfound set in the first place
[20:13] <sjust> 20
[20:13] <NaioN> k
[20:15] * asadpanda (~asadpanda@ has joined #ceph
[20:15] <pmjdebruijn> gregaf: awesome...
[20:15] <pmjdebruijn> gregaf: thanks!
[20:16] <NaioN> sjust: waiting for them to come online again
[20:16] <sagewk> sid build is broken... we need a gitbuilder for that
[20:17] <NaioN> both up again and quering 54
[20:18] <NaioN> the log is filling fast, can I grep on something?
[20:22] <sjust> NaioN: if it's still stuck, can you grab the two logs from the restart until now and zip them up?
[20:23] <sjust> and dump them to cephdrop@ceph.com
[20:24] <NaioN> hmmm for your information the cluster is also recovering so there's a lot of traffic! :)
[20:24] <sjust> yeah
[20:24] <NaioN> I will restart them both with clean logs
[20:25] <sjust> ok
[20:30] <NaioN> do you accept files bigger than 10M?
[20:30] <NaioN> one is 8M and one is 12M
[20:31] <NaioN> or is it ftp or something?
[20:31] <CristianDM> gregaf: Yes. I upgrade now to 0.49
[20:31] <CristianDM> ceph version 0.49 (commit:ca6265d0f4d68a5eb82b5bfafb450e8e696633ac)
[20:33] <gregaf> CristianDM: and what's the output of "ceph -s"?
[20:33] <sjust> NaioN: it's sftp, both should be fnie
[20:33] <NaioN> oh ok
[20:34] <CristianDM> unrecognized subsystem
[20:34] <NaioN> sjust: password?
[20:35] <NaioN> sorry didn't use sftp :)
[20:35] <sjust> ah, ok
[20:35] <NaioN> euhmm do you use a password?
[20:37] <gregaf> CristianDM: can you copy and paste the whole shell command and output, please?
[20:37] <NaioN> uploading...
[20:38] <gregaf> CristianDM: because I don't see the word "subsystem" anywhere that it could turn up in your output
[20:38] <NaioN> sjust: done...
[20:40] <gregaf> CristianDM: in fact that looks to me like your monitors (or at least one of them) are still running an older version (0.47 or earlier)
[20:40] <CristianDM> mmm
[20:40] <CristianDM> And how I can check it
[20:40] <CristianDM> ?
[20:40] <CristianDM> This don??t upgrade with the repos'
[20:40] <CristianDM> ?
[20:40] <gregaf> ssh to the machines running monitors and run ceph-mon ???version
[20:40] <gregaf> see what you've got
[20:40] <gregaf> obviously they should upgrade, but something has gone wrong somewhere
[20:40] <gregaf> see if there are multiple binaries installed that could be getting set
[20:41] <gregaf> s/set/used
[20:41] <CristianDM> ceph version 0.47.2 (commit:8bf9fde89bd6ebc4b0645b2fe02dadb1c17ad372)
[20:41] <CristianDM> :P
[20:41] <CristianDM> Wops
[20:41] <gregaf> yes
[20:41] <Tv_> gregaf: eww your im client just em-dashed that command line option
[20:41] <gregaf> are you sure you actually upgraded them?
[20:41] <CristianDM> yes
[20:41] <Tv_> ;)
[20:41] <gregaf> Tv_: yeah, there's not a great way to turn off the system-wide spell-checking in specific applications
[20:41] <gregaf> and whenever you aren't typing command-line options I want that conversion :(
[20:42] <CristianDM> gregaf: Know how I can force the upgrade?
[20:43] <gregaf> CristianDM: well, I'd start by seeing if the packages are actually new or not...
[20:44] <CristianDM> ceph install
[20:44] <CristianDM> ceph-common install
[20:44] <CristianDM> ceph-fuse install
[20:44] <CristianDM> gceph install
[20:44] <CristianDM> libcephfs1 install
[20:44] <gregaf> that's the only problem I can think of, and I'm not much of a sysadmin but I imagine you'll want some variation of checking out what's available with apt-cache, trying an apt-get install ceph and seeing what version it selects, and then trying it by manually specifying the version
[20:44] <gregaf> yes, you want to check out the package versions, not just their existence
[20:45] <CristianDM> I will force reinstall
[20:46] <gregaf> okay, back later, lunch time
[20:46] <CristianDM> WOW
[20:47] <CristianDM> Now works. ... I don??t know why apt-get don??t upgrade this packages
[21:14] <CristianDM> gregaf: All work now. Currently ceph is working. When upgrade the mon from 0.47.2 to 0.49 this need to upgrade the data?
[21:15] * maelfius (~Adium@ Quit (Quit: Leaving.)
[21:19] * NashTrash (~Adium@ has joined #ceph
[21:19] <NashTrash> Hello Ceph'ers
[21:31] * CristianDM (~CristianD@ Quit ()
[21:34] <NashTrash> Has anyone been successful running a schroot from a ceph mounted director?
[21:35] <NashTrash> When I try to enter into the schroot environment the process just hangs
[21:36] <joshd> does schroot use special syscalls or ioctls? an strace of that happening might be interesting
[21:37] <NashTrash> Unfortunately, my knowledge of how schroot works is zero, as is my knowledge of strace
[21:37] <NashTrash> I am happy to try anything to figure this out though
[21:40] <joshd> just run strace -f -o/path/to/non-cephfs-file schroot [args]
[21:41] <NashTrash> Ok, one moment...
[21:46] <NashTrash> Ok. I have a 173k strace file now. How much of it do you want? The last line is:
[21:46] <NashTrash> stat("/var/lib/schroot/mount/scm-6144dd60-950e-4806-af1c-4c7b676eb150/etc/resolv.conf",
[21:46] <NashTrash> and nothing after the comma
[21:46] <dmick> sounds like an insufficiently configured root env, then
[21:47] <dmick> does schroot <trivial command> work? Like, say, id?
[21:49] <NashTrash> Nope. I get Chroot setup failed: stage=setup-start
[21:50] <dmick> so this doesn't involve Ceph, most likely. chroot envs can be tricky.
[21:51] <NashTrash> It seems to work if I use a real drive to host the schroot but move it to ceph and it does not
[21:51] <Tv_> dmick: why would the stat hang?
[21:52] <Tv_> NashTrash: cat you "cat /var/lib/schroot/mount/scm-6144dd60-950e-4806-af1c-4c7b676eb150/etc/resolv.conf" outside the chroot
[21:52] <Tv_> *can you
[21:52] <NashTrash> Hmm??? Operation not permitted
[21:52] <Tv_> NashTrash: sudo cat ...
[21:53] <NashTrash> Still not permitted
[21:53] <NashTrash> I now also see a process "cp --preserve=all /etc/resolv.conf /var/lib/schroot/mount/scm-6dd130fb-f253-412c-889c-a710f5844e89/etc/resolv.conf" in ps. Looks like it has hung
[21:54] <Tv_> NashTrash: that sounds like a cephfs bug, or perhaps the osds storing that file are inaccessible, etc
[21:57] <NashTrash> Ok. I can ls the etc/ directory, but I get nothing from a cat of any of the files.
[21:57] <Tv_> NashTrash: "get nothing" or "it hangs"?
[21:57] <Tv_> catting an empty file gets you nothing, successfully
[21:57] <NashTrash> Get nothing.
[21:57] <NashTrash> ls -al shows files with non-zero sizes, but they all cat nothing.
[21:58] <Tv_> NashTrash: can you pastebin a strace of that cat?
[21:58] * EmilienM (~EmilienM@ Quit (Remote host closed the connection)
[21:59] <Tv_> (try explaining that sentence to your mother...)
[21:59] * EmilienM (~EmilienM@ has joined #ceph
[22:02] <NashTrash> Now it hangs...
[22:03] <Tv_> NashTrash: is the ceph cluster healthy?
[22:03] <NashTrash> As far as I knew a bit ago. Let me check...
[22:05] <sjust> NaioN: it appears to be an actual bug, working on a patch now
[22:05] <sjust> thanks for the logs!
[22:05] <NashTrash> Cephs looks very happy. Status shows HEALTH_OK
[22:12] <NaioN> sjust: Nice, thanks!
[22:15] <NaioN> just drop a note here if it's ready and where I can find it (mailinglist I assume) and we will try it and let you know if it works!
[22:15] <sjust> ok
[22:17] * The_Bishop (~bishop@2a01:198:2ee:0:346b:e850:d17:ac41) Quit (Quit: Wer zum Teufel ist dieser Peer? Wenn ich den erwische dann werde ich ihm mal die Verbindung resetten!)
[22:18] <NashTrash> Tv_: I have tried a couple of other things and they all lock up. Are there other routes I can pursue on this or should I just move on?
[22:19] <Tv_> NashTrash: log files, starting from the mdses..
[22:21] <NashTrash> Hmm??? ceph-mds.0.log is completely (zero bytes) empty
[22:24] <NashTrash> Ah, now I see. ceph-mds.2.log is good. Guess having 3 mds's can lead to this.
[22:24] * mpw (~mpw@chippewa-nat.cray.com) Quit (Quit: Nettalk6 - www.ntalk.de)
[22:24] <NashTrash> Here are the last 1000 lines from the mds (basically all of my testing today): http://pastebin.com/19E15BTW
[22:27] <NashTrash> There are a ton of ms_handle_connect and ms_handle_reset among the 3 mds
[22:35] * nhorman (~nhorman@2001:470:8:a08:7aac:c0ff:fec2:933b) Quit (Quit: Leaving)
[22:36] <sjust> NaioN: wip_might_have_unfound should take care of that problem
[22:43] * eightyeight (~atoponce@pinyin.ae7.st) has joined #ceph
[22:48] * EmilienM (~EmilienM@ Quit (Quit: Leaving...)
[22:50] <dspano> NashTrash: Were you able to figure out what went wrong? I'm just curious in case I run into something similar.
[23:42] * Leseb (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) has joined #ceph
[23:53] * The_Bishop (~bishop@p4FCDF575.dip.t-dialin.net) has joined #ceph
[23:55] * dspano (~dspano@rrcs-24-103-221-202.nys.biz.rr.com) Quit (Quit: Leaving)
[23:59] <Tv_> oh nice, libreoffice just ate a few dozen slides.. powerpoint conversion suck
[23:59] * NashTrash (~Adium@ Quit (Quit: Leaving.)

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.