#ceph IRC Log


IRC Log for 2010-07-21

Timestamps are in GMT/BST.

[0:07] <sagewk> wido: added some items to ceph.conf to control default pool parameters
[0:17] * deksai (~chris@71-13-57-82.dhcp.bycy.mi.charter.com) Quit (Quit: Leaving.)
[0:46] <darkfader> sagewk: i found an issue where ls -l on one subdirectory will get stuck. my ceph is quite current (2 days ago) but the linux kernel is the one from debian squeeze. should i rather upgrade the kernel first or report somewhere already?
[0:46] <sagewk> report.. i dont remember a kernel bug that causes that off the top of my head
[0:47] <sagewk> cat /proc/sys/kernel/ceph/*/mdsc
[0:48] <sagewk> single or multiple mds?
[0:55] <darkfader> hi sagewk
[0:55] <darkfader> a) i don't have the proc pieces and don't know why'
[0:56] <darkfader> 2 mds's in my config, right now i suspect they're unhappy:
[0:56] <darkfader> mds e70: 1/1/1 up {0=up:active}, 2 up:standby(laggy or crashed)
[0:56] <sagewk> oops, no /porc, just /sys/kernel/debug/ceph/*/mdsc
[0:56] <darkfader> i can access 99% of the data though
[0:56] <darkfader> waxu0026:/tmp# ls /sys/kernel/debug/ceph/*/mdsc
[0:56] <darkfader> ls: cannot access /sys/kernel/debug/ceph/*/mdsc: No such file or directory
[0:56] <darkfader> no better
[0:56] <sagewk> mount -t debugfs none /sys/kernel/debug
[0:56] <darkfader> thank you
[0:56] <darkfader> /sys/kernel/debug/ceph/88b6bd47-e483-ed14-6f50-d816679f3171.client6524/mdsc
[0:57] <sagewk> cat that file?
[0:58] <darkfader> it's 0 bytes
[0:58] <darkfader> it shouldn't be, huh?
[0:58] <darkfader> i have ceph-dbg installed though. is there some extra module to load?
[0:58] <sagewk> hmm, and the 'ls' is still hung?
[0:59] <sagewk> it'll show as 0 bytes in ls, but you can still cat it, btw
[0:59] <darkfader> wait a sec i need to make a new ls -l :)
[1:00] <darkfader> 10675 mds0 getattr #10000007025
[1:00] <darkfader> 10676 mds0 getattr #10000007025
[1:00] <sagewk> ok, then 'ceph mds dumpcache /tmp/foo' and grep 10000007025 /tmp/foo
[1:01] <sagewk> er, ceph mds tell 0 dumpcache /tmp/foo
[1:02] <darkfader> it ends with child dirty
[1:02] <darkfader> just a sec
[1:02] <darkfader> http://wartungsfenster.pastebin.org/409680
[1:02] <darkfader> (ignore the path, i needed mixed-size test data)
[1:04] <sagewk> the problem is '(ifile lock->sync w=1)'.. you don't by chance have mds logging turned up?
[1:04] <darkfader> i'll add a cehp -w posting in a moment. i think i should have 5 osds up, so there's something odd here.
[1:04] * pcish (dc8729c2@ircip1.mibbit.com) Quit (Ping timeout: 480 seconds)
[1:04] <darkfader> no, not atm
[1:05] <darkfader> is that the debug ms = 1 setting in cehp.conf?
[1:05] <sagewk> and debug mds = 20
[1:05] <darkfader> okay will do.
[1:05] <sagewk> well no worries, restarting the mds will unwedge it (though it won't tell us why it wedged)
[1:06] <sagewk> unless the problem is the osds preventing an mds request from completing.. what's ceph -s say?
[1:07] <darkfader> after restarting it all it won't come up just yet
[1:07] <darkfader> iif you reload the pastebin post, i have amended the ceph -w
[1:08] <darkfader> about 50% started on restart, i'll manually start whats missing.
[1:09] <darkfader> http://wartungsfenster.pastebin.org/409699
[1:09] <darkfader> one osd (-i 3) is not coming up.
[1:10] <darkfader> and there's many lines of debug now :)
[1:13] <darkfader> you're right, ls and du all snappy fast now
[1:13] <darkfader> sorry i had it turned down in the first place
[1:14] <darkfader> it was hard to get a basic grasp of the messages / status while debug was on, so i turned had turned it off while doing first steps
[1:15] <darkfader> btw i love how it already perfectly resyncs on osd loss
[1:16] <darkfader> http://wartungsfenster.pastebin.org/409720 <- after the resync is done, one mds got blacklisted. is that because i had an even number?
[1:17] <sagewk> that happens to any cmds instance that is stopped/crashed. it makes sure any unresposnisve process doesn't issues stray writes etc.
[1:18] <darkfader> but should it be stopped?
[1:19] <sagewk> you restarted the mds.. so the old instance got blacklisted. totally normal. the blacklisting will expire on its own.
[1:19] <darkfader> oh, cool. got it :)
[1:20] <darkfader> brb, shower. i'll try to break things again so i can come back with debug data :)
[1:21] <darkfader> btw, do you need/want some infiniband hardware for testing?
[1:22] <sagewk> hmm, is the proper response to "do you need/want some infiniband hardware" not always "yes"?
[1:23] <yehudasa> as long as you strike out the 'need'
[1:23] <sagewk> we probably wouldn't have time to use it just yet. that probably answers your question... :)
[2:10] * ghaskins_mobile (~ghaskins_@66-189-114-103.dhcp.oxfr.ma.charter.com) has joined #ceph
[2:16] * tjikkun (~tjikkun@2001:7b8:356:0:204:bff:fe80:8080) Quit (Ping timeout: 480 seconds)
[2:28] * eternaleye (~quassel@184-76-53-210.war.clearwire-wmx.net) Quit (Ping timeout: 480 seconds)
[2:31] * ghaskins_mobile (~ghaskins_@66-189-114-103.dhcp.oxfr.ma.charter.com) Quit (Quit: Leaving)
[2:42] * tjikkun (~tjikkun@2001:7b8:356:0:204:bff:fe80:8080) has joined #ceph
[2:45] * eternaleye (~quassel@184-76-53-210.war.clearwire-wmx.net) has joined #ceph
[3:08] * henrycc (~henry_c_c@59-124-35-221.HINET-IP.hinet.net) Quit (Server closed connection)
[3:09] * henrycc (~henry_c_c@59-124-35-221.HINET-IP.hinet.net) has joined #ceph
[3:29] * deksai (~chris@71-13-57-82.dhcp.bycy.mi.charter.com) has joined #ceph
[3:39] * deksai (~chris@71-13-57-82.dhcp.bycy.mi.charter.com) has left #ceph
[3:51] * Osso (osso@AMontsouris-755-1-18-109.w90-46.abo.wanadoo.fr) Quit (Ping timeout: 480 seconds)
[5:45] * ghaskins_mobile (~ghaskins_@66-189-114-103.dhcp.oxfr.ma.charter.com) has joined #ceph
[6:51] * f4m8_ is now known as f4m8
[8:45] * eternaleye (~quassel@184-76-53-210.war.clearwire-wmx.net) Quit (Ping timeout: 480 seconds)
[9:28] * allsystemsarego (~allsystem@ has joined #ceph
[10:20] * henrycc (~henry_c_c@59-124-35-221.HINET-IP.hinet.net) Quit ()
[12:58] * tjikkun (~tjikkun@2001:7b8:356:0:204:bff:fe80:8080) Quit (Ping timeout: 480 seconds)
[13:17] * Yoric (~David@ has joined #ceph
[13:17] <Yoric> hi
[13:36] * Osso (~osso@ has joined #ceph
[13:47] * Osso (~osso@ Quit (Ping timeout: 480 seconds)
[13:55] <wido> hi
[13:56] * Osso (osso@AMontsouris-755-1-7-189.w86-212.abo.wanadoo.fr) has joined #ceph
[13:56] <andret> hi
[13:57] <andret> is there something available for monitoring ceph with nagios? any api or /proc
[14:10] * ghaskins_mobile (~ghaskins_@66-189-114-103.dhcp.oxfr.ma.charter.com) Quit (Quit: Leaving)
[14:13] * ghaskins_mobile (~ghaskins_@66-189-114-103.dhcp.oxfr.ma.charter.com) has joined #ceph
[14:27] <wido> andret: what would you like to monitor?
[14:27] <wido> i'm using Zabbix myself, just checking the amount of cmon, cmds and cosd processes
[14:28] <wido> also the checksum of /etc/ceph/ceph.conf and the free diskspace in /var/log/ceph
[14:28] <wido> that's it for now
[14:28] <wido> i'm afk
[14:38] <andret> simply if it is working or if a human should do something
[14:39] <andret> so it is the output of ceph mon stat
[14:39] <andret> or something similar
[14:39] <andret> where do you see the free diskspace?
[14:54] <darkfader> wido: i also think there should be a way to monitor the amont of "stale" chunks
[14:55] <darkfader> process monitoring we probably all got
[14:56] <darkfader> i wanna do some snmp trap stuff for ceph, but some api might really be useful
[14:57] <darkfader> or for example mds status - the process is running, but is the status healtly or sad
[14:59] <darkfader> andret: i think it will be hard to build a monitor based on the stat output
[15:00] <darkfader> but it seems best to hook into the mon
[15:23] <andret> oh, i have just killed two of three nodes in my testbed and the mountpoint was stale (what a surprise ;)
[15:24] <andret> unfortunately it did not come back propperly after booting the other nodes.
[15:32] <darkfader> for me most of the time it's just some daemons not running
[15:33] <darkfader> are they all there?
[15:34] <andret> yes, even a restart of ceph did node, too
[15:34] <andret> did not help, i mean
[15:34] <andret> sorry, it is far to hot in my office
[15:37] <darkfader> np, it's no better here
[15:37] <darkfader> ceph -w will just hang, too? or just the fs access?
[15:37] <darkfader> fs hang usually means mds problem as far as I understand this
[15:48] <wido> darkfader: ofcourse, extra monitoring would be really nice
[15:48] <wido> librados has some features to read the diskspace usage, etc, etc, but there is no implementation right now
[15:48] <andret> next one: dd if=/dev/zero of=nullfile bs=1024
[15:48] <andret> cephbackup2 kernel: [ 1586.862989] Oops: 0000 [#1] SMP
[15:48] <andret> cephbackup2 kernel: [ 1586.863111] last sysfs file: /sys/devices/virtual/block/loop7/removable
[15:48] <andret> lol
[15:48] <wido> but some monitoring would be possible, although not at the moment (i think)
[15:49] <darkfader> will see, if i manage to switch jobs then i should have time for it
[15:50] <darkfader> and some basic monitoring should be not too hard
[15:50] <darkfader> so librados already has a lot?
[15:50] <darkfader> that sounds good
[15:50] <darkfader> s/lot/some/ :)
[15:55] <wido> yes, check out the header file
[16:13] * tjikkun (~tjikkun@2001:7b8:356:0:204:bff:fe80:8080) has joined #ceph
[16:20] * deksai (~chris@dsl093-003-018.det1.dsl.speakeasy.net) has joined #ceph
[16:54] <darkfader> I've added a monitoring page in the wiki, maybe andret could add what he thinks that needs to be monitored
[17:39] * johnl (~johnl@cpc3-brad19-2-0-cust563.barn.cable.virginmedia.com) has joined #ceph
[18:17] * Osso (osso@AMontsouris-755-1-7-189.w86-212.abo.wanadoo.fr) Quit (Quit: Osso)
[18:22] * eternaleye (~quassel@184-76-53-210.war.clearwire-wmx.net) has joined #ceph
[18:31] * Yoric (~David@ Quit (Quit: Yoric)
[18:35] * sagelap (~sage@static-66-14-234-139.bdsl.verizon.net) has joined #ceph
[18:37] * pcish (7a74347d@ircip1.mibbit.com) has joined #ceph
[18:55] <wido> gregaf: the S3 gateway doesn't support partial content yet, right?
[18:57] <yehudasa> wido: partial content?
[18:57] <yehudasa> like reading partial object?
[18:58] <wido> yes
[18:59] <wido> i uploaded a video to the gateway then tried skipping through it with VLC
[18:59] <wido> that doesn't work
[18:59] <yehudasa> it does support S3 content range
[18:59] <wido> content range, that was what i was looking for
[19:01] <wido> VLC does that when you click in the timeline of a video, but it stalls, then starts playing from the beginning
[19:01] <yehudasa> hmm.. I don't know what VLC specifies in the protocol
[19:01] <wido> i'll do a TCP dump and see what it does
[19:01] <yehudasa> the S3 gateway supports HTTP_RANGE
[19:02] <wido> at first i thought it wasn't implemented in the gateway, but when you say it does, i'll dig in some deeper
[19:02] <yehudasa> I actually found some bug in it yesterday
[19:02] <yehudasa> but it should be fixed now
[19:02] <yehudasa> but it might be that the support is partial\
[19:05] <wido> Range: bytes=43294261-\r\n
[19:05] <wido> that is the header which VLC is sending
[19:05] <yehudasa> hmm
[19:05] <yehudasa> do you have the apache log?
[19:06] <wido> yes, sure
[19:06] <wido> error or acces?
[19:06] <yehudasa> error
[19:06] * sagelap (~sage@static-66-14-234-139.bdsl.verizon.net) Quit (Ping timeout: 480 seconds)
[19:07] <wido> http://www.pastebin.org/410959
[19:07] <yehudasa> actually I need the part before that
[19:07] <yehudasa> like where it parses the header
[19:08] <yehudasa> it should dumps all the variables
[19:08] <wido> ok
[19:08] <wido> one sec
[19:10] <wido> http://www.pastebin.org/410962
[19:10] <wido> i truncated the error log and did the test again
[19:11] <wido> it seems to me that only the first request comes through and when i click in the timeline, it does not result in a HTTP request
[19:11] <yehudasa> hmm
[19:11] <wido> do you have VLC yourself? The ACL is on public-read
[19:12] <yehudasa> I probably have one
[19:12] <yehudasa> yeah
[19:13] <wido> could it be FastCGI which is blocking the second request?
[19:13] <yehudasa> so the VLC actually sends a second request but you don't see it in the apache logs?
[19:14] <wido> i'm not sure
[19:14] <wido> just tested with a regular Apache server where the same file is, there clicking through the timeline works
[19:14] <wido> i see the requests with Wireshark
[19:15] <wido> hmm, i see some internal servers errors in the access log
[19:15] <yehudasa> oh, ok
[19:15] <wido> might be the gateway who is not responding to FastCGI?
[19:16] <yehudasa> usually I get internal server error when the fastcgi sends some bad formatted response
[19:16] <yehudasa> or a bad error code
[19:16] <wido> or it doesn't respond at all
[19:18] <wido> sagewk: the monitor crashes i was seeing were due to lack of free diskspace i assume. Some last_commited files were zero in filesize. Today i added some disks to the monitors and placed their data on a seperate partition, no more monitor crashes
[19:47] * johnl (~johnl@cpc3-brad19-2-0-cust563.barn.cable.virginmedia.com) Quit (Quit: bye)
[21:04] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) has joined #ceph
[21:05] * Osso (osso@AMontsouris-755-1-7-189.w86-212.abo.wanadoo.fr) has joined #ceph
[21:05] * eternaleye (~quassel@184-76-53-210.war.clearwire-wmx.net) Quit (Ping timeout: 480 seconds)
[21:06] * Osso_ (osso@AMontsouris-755-1-7-189.w86-212.abo.wanadoo.fr) has joined #ceph
[21:06] * Osso (osso@AMontsouris-755-1-7-189.w86-212.abo.wanadoo.fr) Quit (Read error: Connection reset by peer)
[21:06] * Osso_ is now known as Osso
[21:42] <wido> "7fea0d6fb710 client7411.objecter pg 19.a7e9 on [] is laggy: 705" does that mean that osd19 is responding slow?
[21:45] <wido> i'm trying to upload a 5.3MB file through the RADOS gateway, whole cluster is idle, load is low on all the nodes
[21:58] * sagelap (~sage@ip-66-33-206-8.dreamhost.com) has joined #ceph
[22:02] * sagelap (~sage@ip-66-33-206-8.dreamhost.com) Quit ()
[22:04] <sagewk> wido: that means pool 19 (see 'ceph osd dump -o -' output) pg a7e9 maps to no osds ([]).. probably lots of nodes down?
[22:05] <sagewk> 'ceph pg map 19.a7e9' to see where that pg should (currently) map to
[22:38] <yehudasa> wido: where did you see that range: bytes=432.. message?
[22:39] <yehudasa> I'm trying to reproduce it and vlc doesn't send anything
[22:41] <yehudasa> what vlc version are you using?
[22:47] * eternaleye (~quassel@184-76-53-210.war.clearwire-wmx.net) has joined #ceph
[23:02] * allsystemsarego (~allsystem@ Quit (Quit: Leaving)
[23:19] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) Quit (Quit: Yoric)
[23:21] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) has joined #ceph
[23:30] * deksai (~chris@dsl093-003-018.det1.dsl.speakeasy.net) Quit (Quit: Leaving.)

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.