#ceph IRC Log


IRC Log for 2011-10-08

Timestamps are in GMT/BST.

[0:01] <sjust> df__: it looks like a filesystem operation hung
[0:02] <df__> that was on vc-fs3, affecting osd.2?
[0:03] <sjust> on fs2, it looks like
[0:03] <df__> time?
[0:03] <sjust> between 09:54:09 and 10:32:06
[0:04] <sjust> most likely at 10:31:06
[0:05] <sjust> the disk_tp timed out message indicates that the watcher thread detected that thread 7f5553059700 hung
[0:06] <df__> hmm, it will still be in that state now perhaps. is there anything i can poke to determine what it is hung on?
[0:07] <df__> the process isn't blocking for long periods such that the kernel scheduler moans about it
[0:07] <sjust> I was hoping to get a hint from the earlier logs. The operation it got stuck on will be in the journal, so you could restart the OSD with filestore debugging at 20
[0:08] <df__> ok, config stanza for that?
[0:09] <sjust> in the osd section: debug filestore = 20
[0:15] <df__> ftp://ftp.kw.bbc.co.uk/davidf/priv/ceph/aiChae4p.txt
[0:26] <df__> sjust, btw, is there a command to adjust the configuration of an osd without restarting?
[0:29] <sjust> ceph osd tell '(mds name)' injectargs '--debug_filestore 20' I think
[0:29] <sjust> oops
[0:29] <sjust> ceph osd tell osd.# injectargs '--debug_filestore 20' I think
[0:32] <df__> failed again, just updating that log
[0:34] <df__> ok, see the same file, should run up to about 22:30
[0:38] <df__> right, i must head to bed (it is actually GMT+1 here). if there is any further information i can collect (or a suggested fix), i'll check scrollback in the morning
[0:39] <df__> sjust, thank you for your help so far
[0:39] <sjust> ok, thanks
[1:04] * verwilst (~verwilst@dD576F54C.access.telenet.be) Quit (Quit: Ex-Chat)
[1:15] * bchrisman (~Adium@ Quit (Quit: Leaving.)
[1:19] * jojy (~jojyvargh@ Quit (Quit: jojy)
[3:09] * joshd (~joshd@aon.hq.newdream.net) Quit (Quit: Leaving.)
[3:12] * bugoff (bram@november.openminds.be) Quit (Read error: Connection reset by peer)
[3:12] * bugoff (bram@november.openminds.be) has joined #ceph
[4:20] * votz (~votz@pool-108-52-121-23.phlapa.fios.verizon.net) Quit (Quit: Leaving)
[4:26] * Nightdog (~karl@190.84-48-62.nextgentel.com) Quit (Remote host closed the connection)
[4:36] * bchrisman (~Adium@c-98-207-207-62.hsd1.ca.comcast.net) has joined #ceph
[5:11] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[5:17] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[7:34] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[7:35] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[12:11] * verwilst (~verwilst@dD57690BD.access.telenet.be) has joined #ceph
[12:48] * Dantman (~dantman@S010600259c4d54ff.vs.shawcable.net) Quit (Read error: Connection reset by peer)
[12:50] * Nightdog (~karl@190.84-48-62.nextgentel.com) has joined #ceph
[12:58] * verwilst (~verwilst@dD57690BD.access.telenet.be) Quit (Quit: Ex-Chat)
[13:13] * Dantman (~dantman@S010600259c4d54ff.vs.shawcable.net) has joined #ceph
[14:02] * adjohn (~adjohn@ has joined #ceph
[15:47] * mtk (~mtk@ool-182c8e6c.dyn.optonline.net) Quit (Ping timeout: 480 seconds)
[16:32] * adjohn (~adjohn@ Quit (Quit: adjohn)
[18:23] * jward (~jward@188-220-42-49.zone11.bethere.co.uk) has joined #ceph
[20:36] * jward (~jward@188-220-42-49.zone11.bethere.co.uk) Quit (Quit: Leaving)
[21:49] * votz (~votz@pool-108-52-121-23.phlapa.fios.verizon.net) has joined #ceph

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.