#ceph IRC Log


IRC Log for 2011-04-05

Timestamps are in GMT/BST.

[0:04] * djlee (~dlee064@des152.esc.auckland.ac.nz) has joined #ceph
[0:13] * st-13176 (~st-13176@a89-154-147-132.cpe.netcabo.pt) has joined #ceph
[0:13] * st-13176 (~st-13176@a89-154-147-132.cpe.netcabo.pt) Quit ()
[0:14] * st-13189 (~st-13189@a89-154-147-132.cpe.netcabo.pt) has joined #ceph
[0:15] * st-13189 (~st-13189@a89-154-147-132.cpe.netcabo.pt) Quit ()
[0:17] <cmccabe1> do you guys know anything about cephbeta.dreamhost.com?
[0:17] <gregaf> it's the rgw for the playground?
[0:17] <cmccabe1> well, it seems to be completely unresponsive now
[0:18] <cmccabe1> fyi
[0:21] <Tv> so far, nobody's admitted any operative knowledge of the installation..
[0:22] <gregaf> yeah, Sage and Yehuda have always handled it
[0:22] <Tv> sage is on it
[0:22] <gregaf> one of the admins might know more, I know it got redirected at some point
[0:22] <sjust> ballgate0
[0:23] <gregaf> I'm not sure if it still is…?
[0:23] <sjust> it is, I think
[0:23] <sjust> at least, there seems to be a malfunctioning rgw installation running there
[0:23] <gregaf> it was originally but I think it might have changed when Yehuda switched to the new code
[0:23] <gregaf> or maybe he just upgraded ballgate0
[0:24] <sjust> [Mon Apr 04 21:21:41 2011] [error] [client] FastCGI: incomplete headers (0 bytes) received from server "/home/cephbeta/cephbeta.dreamhost.com/s3gw.fcgi"
[0:24] <sjust> [Mon Apr 04 21:22:52 2011] [error] [client] FastCGI: comm with (dynamic) server "/home/cephbeta/cephbeta.dreamhost.com/s3gw.fcgi" aborted: (first read) idle timeout (60 sec)
[0:24] <sjust> [Mon Apr 04 21:22:52 2011] [error] [client] FastCGI: incomplete headers (0 bytes) received from server "/home/cephbeta/cephbeta.dreamhost.com/s3gw.fcgi"
[0:24] <sjust> [Mon Apr 04 21:23:53 2011] [error] [client] FastCGI: comm with (dynamic) server "/home/cephbeta/cephbeta.dreamhost.com/s3gw.fcgi" aborted: (first read) idle timeout (60 sec)
[0:24] <sjust> [Mon Apr 04 21:23:53 2011] [error] [client] FastCGI: incomplete headers (0 bytes) received from server "/home/cephbeta/cephbeta.dreamhost.com/s3gw.fcgi"
[0:24] <sjust> [Mon Apr 04 21:25:04 2011] [error] [client] FastCGI: comm with (dynamic) server "/home/cephbeta/cephbeta.dreamhost.com/s3gw.fcgi" aborted: (first read) idle timeout (60 sec)
[0:24] <sjust> [Mon Apr 04 21:25:04 2011] [error] [client] FastCGI: incomplete headers (0 bytes) received from server "/home/cephbeta/cephbeta.dreamhost.com/s3gw.fcgi"
[0:24] <sjust> [Mon Apr 04 22:05:24 2011] [error] [client] FastCGI: comm with (dynamic) server "/home/cephbeta/cephbeta.dreamhost.com/s3gw.fcgi" aborted: (first read) idle timeout (60 sec)
[0:24] <sjust> [Mon Apr 04 22:05:24 2011] [error] [client] FastCGI: incomplete headers (0 bytes) received from server "/home/cephbeta/cephbeta.dreamhost.com/s3gw.fcgi"
[0:24] * Terminus (~Terminus@ip-66-33-206-8.dreamhost.com) has joined #ceph
[0:24] <sjust> [Mon Apr 04 22:06:34 2011] [error] [client] FastCGI: comm with (dynamic) server "/home/cephbeta/cephbeta.dreamhost.com/s3gw.fcgi" aborted: (first read) idle timeout (60 sec)
[0:24] <sjust> [Mon Apr 04 22:06:34 2011] [error] [client] FastCGI: incomplete headers (0 bytes) received from server "/home/cephbeta/cephbeta.dreamhost.com/s3gw.fcgi"
[0:24] <sjust> [Mon Apr 04 22:22:56 2011] [error] [client] FastCGI: comm with (dynamic) server "/home/cephbeta/cephbeta.dreamhost.com/s3gw.fcgi" aborted: (first read) idle timeout (60 sec)
[0:24] <sjust> [Mon Apr 04 22:22:56 2011] [error] [client] FastCGI: incomplete headers (0 bytes) received from server "/home/cephbeta/cephbeta.dreamhost.com/s3gw.fcgi"
[0:24] <sjust> sorry for the spam
[0:26] * sagelap (~sage@ has joined #ceph
[0:26] <sagelap> playground's problem is osd3 failing to authorized connections
[0:26] <sagelap> 2011-04-04 15:25:41.986239 7ff67f9f8910 auth: could not find secret_id=2785
[0:26] <sagelap> 2011-04-04 15:25:41.986254 7ff67f9f8910 cephx: verify_authorizer could not get s
[0:26] <sagelap> ervice secret for service osd secret_id=2785
[0:26] <sagelap> 2011-04-04 15:25:41.986267 7ff67f9f8910 osd3 27701 OSD::ms_verify_authorizer nam
[0:27] <sjust> ah
[0:29] * Terminus is now known as mwodrich
[0:31] <mwodrich> if it helps at all, this is what I'm getting from CrossFTP while trying to connect to cephbeta.dreamhost.com: [R1] CloudFront Error: Request failed with CloudFront Service error (CloudFrontServiceException: httpResponseCode=403, errorType=Sender, errorCode=InvalidClientTokenId, errorMessage=The security token included in the request is invalid, errorDetail=null, errorRequestId=3a76e0b7-5f01-11e0-999a-1dec20947e89 )
[0:31] <mwodrich> and I'm certain that the credentials I'm supplying are correct
[0:32] <cmccabe1> I can't even connect at all, the command-line client just hangs
[0:32] <cmccabe1> I guess I didn't use wireshark, but it sure seems like connect is timing out.
[0:32] <Tv> mwodrich: you should not see an external 403 just because of internal problems
[0:32] <mwodrich> the message perl lib Amazon::S3 spits back for attempting to create a bucket is: 500 Server closed connection without sending any data back
[0:33] <cmccabe1> mwodrich: that's probably closer to the truth
[0:33] <Tv> mwodrich: if that is really connected (your problem goes away once we fix the internal problems), please file a ticket
[0:33] <mwodrich> ok
[0:33] <Tv> mwodrich: ah the 500 looks right
[0:33] <Tv> mwodrich: what did the 403 come from?
[0:33] <cmccabe1> tv: a badly programmed s3 client perchance?
[0:33] <mwodrich> that is the error output from CrossFTP
[0:35] <Tv> well, i don't see InvalidClientTokenId in the ceph source tree at all..
[0:35] <cmccabe1> tv: we don't support cloudfront either
[0:35] <Tv> yeah no clue what CrossFTP / mwodrich is actually doing
[0:35] * ghaskins_mobile (~ghaskins_@66-189-113-47.dhcp.oxfr.ma.charter.com) Quit (Quit: This computer has gone to sleep)
[0:35] <Tv> that message looks like CrossFTP is feeding cephbeta auth codes to CloudFront
[0:36] <Tv> that's.. not gonna work, ever
[0:36] <Tv> mwodrich: disable CloudFront
[0:36] <mwodrich> I just have CrossFTP set to use cephbeta.dreamhost.com, and it was working on Friday
[0:36] <mwodrich> also, I don't have Cloudfront enabled
[0:36] <mwodrich> there is no reason that should be part of the error message
[0:45] <sagelap> the auth error was on the internal protocol. i suspect the osd didn't update the rotating secret or something.. need to look at the logs in detail. in any case, restarted cosd on osd3, should be okay now?
[0:47] * ghaskins_mobile (~ghaskins_@66-189-113-47.dhcp.oxfr.ma.charter.com) has joined #ceph
[0:47] <mwodrich> yes, things appear to be working now
[0:48] <mwodrich> both with the perl S3 library and CrossFTP
[0:49] <bchrisman> I've got this strange error in my ceph -s output… and perf is grossly low (1-2MB/min): http://pastebin.com/NJazJCeL
[0:53] * aliguori (~anthony@cpe-70-123-132-139.austin.res.rr.com) Quit (Ping timeout: 480 seconds)
[0:53] * allsystemsarego (~allsystem@ Quit (Quit: Leaving)
[0:56] <gregaf> bchrisman: how are you measuring performance?
[0:56] <bchrisman> watching ceph -w
[0:57] * sagelap (~sage@ Quit (Ping timeout: 480 seconds)
[0:57] <bchrisman> over about 45 minutes
[0:57] <gregaf> I mean what's the workload?
[0:57] <bchrisman> cp -a on a build tre
[0:57] <gregaf> oh, 1-2MB/MINUTE?
[0:57] <gregaf> yeesh
[0:57] <bchrisman> yeah
[0:57] <bchrisman> wondering if that error in my ceph -s is the culprit
[0:58] <bchrisman> it's way slower than it was last time I tried the nfs test
[0:58] <bchrisman> (orders of magnitude)
[0:58] <gregaf> well that error means that there's an inode which somehow thinks it has a negative number of files underneath it
[0:58] <bchrisman> what the reference to ~mds0/? Is that basically where the bum data is?
[0:58] <gregaf> I don't think it should cause slowness on its own but something's not working properly
[0:59] <gregaf> oh, I didn't even look at the dir…that's very odd
[0:59] <gregaf> that's the per-MDS data, which includes the MDS log and…not much else, I don't think
[0:59] <gregaf> I wonder if your MDS log got disabled via this issue so it's running without a log
[1:00] <bchrisman> hmm
[1:00] <gregaf> that might be sufficient to destroy write throughput in a cp workload
[1:00] * sagelap (~sage@ has joined #ceph
[1:00] <djlee> gregaf, regarding your recent email response for the slow write, I think x2, x3 I also get the similar performance, and was considered normal
[1:00] <gregaf> but I actually don't even know if that's possible
[1:00] <bchrisman> there a way to tell? mds dump doesn't provide anything interesting on this
[1:00] <gregaf> sagelap: can the MDSes run without a log?
[1:01] <gregaf> bchrisman: not sure, hopefully sage is actually around and will just know
[1:01] <gregaf> sagelap: and if the mds directory (~mds0) inode gets an rfiles underflow could that cause it to run without a log?
[1:02] <gregaf> (the rstat underflow is being reported in clog and cp -a performance is abysmally slow)
[1:04] <sagelap> gregaf: the logging shouldn't affect/be affected by the mds dir
[1:04] * aliguori (~anthony@cpe-70-123-132-139.austin.res.rr.com) has joined #ceph
[1:04] <gregaf> isn't the log located in that dir, though?
[1:04] <sagelap> (there is an inode referncing the log in that dir, but it's only there to expose things to an admin)
[1:04] <sagelap> MDLog doesn't use it at all
[1:04] <gregaf> it was just a thought anyway
[1:04] <sagelap> i missed some of the history.. is the mds not logging, or you're asking if it can be made to not log intentionally?
[1:05] <gregaf> bchrisman's got a cp-a which is going at a few MB per minute
[1:05] <bchrisman> http://pastebin.com/NJazJCeL
[1:05] <gregaf> and clog reported an rfiles underflow on the ~mds0 inode
[1:05] <bchrisman> yeah
[1:05] <sagelap> i doubt those two things are related.. unless the clog itself is slowing things down
[1:05] <gregaf> the log was just a WAG on my part
[1:06] <sagelap> i'd tail the mds log to see what's up. maybe there is lock thrashing or unnecessary flushes or something
[1:07] <bchrisman> will check
[1:07] <gregaf> yeah, we can go through it properly but I was just trying to figure out if they could be connected and get a hint there
[1:09] <bchrisman> hmm.. nothing but connect/resets in mds tail… about 10 or so in 30 minutes
[1:10] <gregaf> ah, logging pretty low?
[1:10] <bchrisman> I'd think there'd be more activity.. but then again.. it's slow: http://pastebin.com/ZXyzVgw3
[1:11] <bchrisman> yeah.. I haven't upped debug log
[1:12] <gregaf> wait, what's the printout of the dirs at the top?
[1:12] <gregaf> is that part of a printout that's going on for some reason?
[1:12] <bchrisman> hmm
[1:13] <gregaf> or is it normal logging and the MDS really isn't doing anything?
[1:13] <bchrisman> there's a flurry of activity there.
[1:13] <bchrisman> but I don't quite understnad what all those messages mean
[1:13] <bchrisman> looking up something in the mds cache?
[1:15] <gregaf> well it's printing a bunch of dentries out for some reason
[1:15] <gregaf> each of those lines is one dentry
[1:15] <gregaf> I don't know why, though — I expect there's a line somewhere telling you what it's for
[1:15] <bchrisman> yeah.. I'm guessing something's gone horribly wrong there… logfile is huge.. though debug mds is 1
[1:16] <bchrisman> sifting through
[1:16] <gregaf> it might be that it recognized the rstat failure and is printing out the subtree to give you a chance to debug it and find out what happened, I think that's the same dir
[1:17] <bchrisman> hmm
[1:17] <bchrisman> I'll hack around that logfile and try to figure out what's blowing up
[1:19] <bchrisman> heh… http://pastebin.com/B4PqCA3Q
[1:20] <bchrisman> maybe a recursive loop going on?
[1:20] <gregaf> there's a good chance you want to see the lines that aren't dentries
[1:20] <gregaf> what's "tail -10000 mds.scale-192-168-98-109.log | grep -v dentry" give you?
[1:20] <bchrisman> yeah.. there are some at the end.. but those stats suggest above that.. it's all dentry..
[1:21] <bchrisman> mainly these: 2011-04-04 21:28:55.898495 7f430ec32710 mds0.cache.dir(606) mismatch between child accounted_rstats and my rstats!
[1:21] <bchrisman> 2011-04-04 21:28:55.903462 7f430ec32710 mds0.cache.dir(606) mismatch between head items and fnode.fragstat! printing dentries
[1:21] <bchrisman> then five minutes after those… ms_handle_reset/ms_handle_connect
[1:22] <gregaf> ah, yep, that's my debug output at work :)
[1:22] <bchrisman> ahhh so what's going on here? :)
[1:22] <gregaf> the recursive statistics are getting broken somehow
[1:22] <bchrisman> mismatch between child and rstats and mismatch between head itesm and fragstat
[1:23] <bchrisman> rstats versus fragstat? is rstat a recursive stat and fragstat is?
[1:24] <gregaf> the relation between all of them is annoying complicated, but each directory maintains an rstat struct that should match up to the sum of its children's accounted_rstat structs (these are the same type of struct, but accounted_rstat means they've been propagated upward and rstat is the live but unpropagated figure)
[1:24] <gregaf> the fragstat keeps some data about the items in each dir fragment, and should match up to the number of live items in the frag, IIRC
[1:24] <bchrisman> ahh
[1:25] <gregaf> this log output is probably related to the rfiles underflow — both are telling you that something is wrong with the rstats
[1:25] <bchrisman> 1 dir -> 1rstat struct?
[1:25] <gregaf> yeah, rstats and accounted_rstats are both members of CDir
[1:25] * ghaskins_mobile (~ghaskins_@66-189-113-47.dhcp.oxfr.ma.charter.com) Quit (Quit: This computer has gone to sleep)
[1:26] <bchrisman> sum(accounted_rstats) supposed to equal rstats (or some fields thereof?)
[1:26] <gregaf> unfortunately it doesn't really tell me what the actual problem is, unless maybe you've got so many damn files that it actually overflowed the counters...
[1:26] <bchrisman> hmm… got a lot of files.. but shouldn't be that many
[1:26] <gregaf> dir.accounted_rstat = sum(child.rstats)
[1:27] <bchrisman> ahh ok
[1:27] <gregaf> or dir.accounted_rstat = foreach child in dentries: child.rstats (if that's clearer)
[1:28] <bchrisman> 1800 files on the share
[1:28] <bchrisman> I'm accessing this via nfs export.. but this is an mds issue it seems..
[1:28] <bchrisman> unless the mds is relying on the client for some part in this game.
[1:28] <gregaf> yeah, I seriously doubt the file count is the problem ;)
[1:29] <gregaf> I don't think the client should have an influence on this, although I'm not really that familiar with how the reexport works
[1:29] <gregaf> something about hashing, heh
[1:30] <bchrisman> yeah. probably shouldn't be able to cause problems on the mds.
[1:30] <bchrisman> any idea what I can do to help out on debug stuff here?
[1:30] <bchrisman> this is latest master as of Friday
[1:30] <bchrisman> was trying out the patch sage put in which should help the NFS issue.
[1:31] <gregaf> well if it were something I were running into I'd try and track down which inode is the cause of the issues and follow it back
[1:31] <gregaf> if there's not enough logging, try and reproduce with more logging...?
[1:31] <gregaf> regarding the throughput issue, we need to figure out what the bottleneck is
[1:32] <gregaf> looking at the messages the client is sending should tell us that
[1:32] <bchrisman> well.. the log file is 1.3GB… might've just slowed down to write that..
[1:32] <bchrisman> ?
[1:33] <gregaf> yeah, it's probably holding the mds_lock which would prevent any forward progress from the client
[1:33] <gregaf> so if your performance stopped at the same time the output started that's all that is
[1:33] <gregaf> I just thought it was at least partly separate based on your description
[1:34] <bchrisman> might have dumped it multiple times or something.
[1:34] <bchrisman> big log
[1:36] <bchrisman> yeah.. looks like it may have been going on the whoel time.. will check
[1:37] <bchrisman> went on over the course of 2 hrs… looks like during the copy
[1:37] <bchrisman> I should look more carefully.
[1:38] <bchrisman> Is that issue something I should look at? or if I set mds debug to 0, will that stuff not get dumped?
[1:39] <gregaf> umm, lemme check where that printout comes from
[1:41] <gregaf> that's part of CDir::check_rstats, where it prints with dout(1)
[1:41] <gregaf> the function's currently called from a few places and it's actually possible it's too expensive even without an error, hmm
[1:42] <bchrisman> is that in mds/CDir.cc?
[1:42] <bchrisman> I don't see it there.
[1:42] <gregaf> CDir.cc line 205 in my checkout
[1:43] <gregaf> it's called from a couple places in MDCache and in CInode.cc as part of the distributed locking code
[1:44] <gregaf> rstat mismatch is a bug that we want to fix, certainly — it keeps track of subdir sizes and if it's broken that's a problem for anybody who wants that data to be useful
[1:44] <gregaf> bbl!
[1:45] <bchrisman> gregaf: okay… if dout(1) doesn't what I think.. I can temporarily bypass by setting mds log to 0? I can recreate this if I have an idea how I'll go about debugging it...
[1:47] <cmccabe1> bchrisman: turning mds logging to 0 will disable those calls to dout(1)
[1:47] <bchrisman> yeah.. cool..
[1:50] * ghaskins_mobile (~ghaskins_@66-189-113-47.dhcp.oxfr.ma.charter.com) has joined #ceph
[1:59] * Juul (~Juul@ Quit (Read error: Operation timed out)
[2:05] * ghaskins_mobile (~ghaskins_@66-189-113-47.dhcp.oxfr.ma.charter.com) Quit (Quit: This computer has gone to sleep)
[2:07] * aliguori (~anthony@cpe-70-123-132-139.austin.res.rr.com) Quit (Ping timeout: 480 seconds)
[2:19] * greglap (~Adium@ip-66-33-206-8.dreamhost.com) Quit (Quit: Leaving.)
[2:25] * Juul (~Juul@ has joined #ceph
[2:26] * Tv (~Tv|work@ip-66-33-206-8.dreamhost.com) Quit (Ping timeout: 480 seconds)
[2:27] * Juul (~Juul@ Quit ()
[2:38] * sagelap (~sage@ Quit (Ping timeout: 480 seconds)
[2:40] * ghaskins_mobile (~ghaskins_@66-189-113-47.dhcp.oxfr.ma.charter.com) has joined #ceph
[2:44] * greglap (~Adium@ has joined #ceph
[2:46] * ghaskins_mobile (~ghaskins_@66-189-113-47.dhcp.oxfr.ma.charter.com) Quit (Quit: This computer has gone to sleep)
[2:46] * joshd1 (~joshd@ip-66-33-206-8.dreamhost.com) Quit (Quit: Leaving.)
[2:51] * bchrisman (~Adium@70-35-37-146.static.wiline.com) Quit (Quit: Leaving.)
[3:43] * greglap (~Adium@ Quit (Read error: Connection reset by peer)
[3:54] * greglap (~Adium@cpe-76-170-84-245.socal.res.rr.com) has joined #ceph
[4:35] * djlee1 (~dlee064@des152.esc.auckland.ac.nz) has joined #ceph
[4:40] * djlee (~dlee064@des152.esc.auckland.ac.nz) Quit (Ping timeout: 480 seconds)
[5:27] * bchrisman (~Adium@c-98-207-207-62.hsd1.ca.comcast.net) has joined #ceph
[5:33] * joshd (~jdurgin@adsl-75-28-69-238.dsl.irvnca.sbcglobal.net) has joined #ceph
[5:47] * neurodrone (~neurodron@cpe-76-180-162-12.buffalo.res.rr.com) Quit (Read error: Connection reset by peer)
[6:37] * cmccabe1 (~cmccabe@c-24-23-254-199.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[7:18] * joshd (~jdurgin@adsl-75-28-69-238.dsl.irvnca.sbcglobal.net) Quit (Quit: Leaving.)
[7:41] * Dantman (~dantman@S0106001eec4a8147.vs.shawcable.net) Quit (Remote host closed the connection)
[8:14] * Dantman (~dantman@S0106001eec4a8147.vs.shawcable.net) has joined #ceph
[8:16] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) has joined #ceph
[8:50] * sagelap (~sage@c-24-6-150-26.hsd1.ca.comcast.net) has joined #ceph
[8:50] * sagelap (~sage@c-24-6-150-26.hsd1.ca.comcast.net) Quit ()
[9:17] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) Quit (Quit: Yoric)
[10:15] * allsystemsarego (~allsystem@ has joined #ceph
[11:25] * Yoric (~David@ has joined #ceph
[11:28] * Yoric_ (~David@ has joined #ceph
[11:28] * Yoric (~David@ Quit (Read error: Connection reset by peer)
[11:28] * Yoric_ is now known as Yoric
[13:33] * st-16052 (~st-16052@a89-154-147-132.cpe.netcabo.pt) has joined #ceph
[13:34] * st-16052 (~st-16052@a89-154-147-132.cpe.netcabo.pt) Quit ()
[13:49] * st-16539 (~st-16539@a89-154-147-132.cpe.netcabo.pt) has joined #ceph
[13:51] * st-16539 (~st-16539@a89-154-147-132.cpe.netcabo.pt) Quit ()
[14:28] * jbdenis (~jbdenis@brucciu.sis.pasteur.fr) has joined #ceph
[14:33] * Yoric_ (~David@ has joined #ceph
[14:33] * Yoric (~David@ Quit (Read error: Connection reset by peer)
[14:33] * Yoric_ is now known as Yoric
[14:38] * ghaskins_mobile (~ghaskins_@66-189-113-47.dhcp.oxfr.ma.charter.com) has joined #ceph
[14:46] * ghaskins_mobile (~ghaskins_@66-189-113-47.dhcp.oxfr.ma.charter.com) Quit (Quit: This computer has gone to sleep)
[14:48] * ghaskins_mobile (~ghaskins_@66-189-113-47.dhcp.oxfr.ma.charter.com) has joined #ceph
[14:49] * ghaskins_mobile (~ghaskins_@66-189-113-47.dhcp.oxfr.ma.charter.com) Quit ()
[15:12] * aliguori (~anthony@cpe-70-123-132-139.austin.res.rr.com) has joined #ceph
[15:29] * neurodrone (~neurodron@dhcp211-163.wireless.buffalo.edu) has joined #ceph
[16:04] * ghaskins_mobile (~ghaskins_@66-189-113-47.dhcp.oxfr.ma.charter.com) has joined #ceph
[16:58] * alexxy[home] (~alexxy@ Quit (Remote host closed the connection)
[17:02] * alexxy (~alexxy@ has joined #ceph
[17:06] * neurodrone (~neurodron@dhcp211-163.wireless.buffalo.edu) Quit (Quit: neurodrone)
[17:08] * neurodrone (~neurodron@dhcp211-163.wireless.buffalo.edu) has joined #ceph
[17:37] * greglap (~Adium@cpe-76-170-84-245.socal.res.rr.com) Quit (Quit: Leaving.)
[17:43] * johnl_ (~johnl@johnl.ipq.co) Quit (Quit: leaving)
[17:52] * greglap (~Adium@ has joined #ceph
[17:52] <stefanha> Is an MDS required if you only use qemu rbd? I don't want to run a ceph file system, just the object store.
[17:57] <greglap> stefanha: nope, the MDS is only required for the Ceph POSIX layer
[17:57] <stefanha> greglap: Excellent, thanks
[18:06] * johnl (~johnl@johnl.ipq.co) has joined #ceph
[18:07] * ghaskins_mobile (~ghaskins_@66-189-113-47.dhcp.oxfr.ma.charter.com) Quit (Quit: This computer has gone to sleep)
[18:33] * alexxy (~alexxy@ Quit (Remote host closed the connection)
[18:33] * neurodrone (~neurodron@dhcp211-163.wireless.buffalo.edu) Quit (Quit: neurodrone)
[18:33] * alexxy (~alexxy@ has joined #ceph
[18:34] * Tv (~Tv|work@ip-66-33-206-8.dreamhost.com) has joined #ceph
[18:48] * greglap (~Adium@ Quit (Ping timeout: 480 seconds)
[18:51] * ghaskins_mobile (~ghaskins_@66-189-113-47.dhcp.oxfr.ma.charter.com) has joined #ceph
[18:55] * cmccabe (~cmccabe@ has joined #ceph
[18:58] * alexxy[home] (~alexxy@ has joined #ceph
[19:00] * joshd (~joshd@ip-66-33-206-8.dreamhost.com) has joined #ceph
[19:01] * alexxy (~alexxy@ Quit (Ping timeout: 480 seconds)
[19:02] * greglap (~Adium@ip-66-33-206-8.dreamhost.com) has joined #ceph
[19:03] * gregaf (~Adium@ip-66-33-206-8.dreamhost.com) has left #ceph
[19:03] * gregaf (~Adium@ip-66-33-206-8.dreamhost.com) has joined #ceph
[19:04] * Yoric (~David@ Quit (Quit: Yoric)
[19:05] * ghaskins_mobile (~ghaskins_@66-189-113-47.dhcp.oxfr.ma.charter.com) Quit (Quit: This computer has gone to sleep)
[19:20] * ghaskins_mobile (~ghaskins_@66-189-113-47.dhcp.oxfr.ma.charter.com) has joined #ceph
[19:22] * ghaskins_mobile (~ghaskins_@66-189-113-47.dhcp.oxfr.ma.charter.com) Quit ()
[19:28] <gregaf> cmccabe: we have to go at 11, they're in our conference room
[19:28] <cmccabe> gregaf: ok
[19:39] * bernard (bernard@november.openminds.be) has joined #ceph
[19:39] * bernard is now known as wonko_be
[19:39] <wonko_be> hey
[19:40] <wonko_be> any way to recreate a lost journal (wrong rm)
[19:40] <wonko_be> or should I recreate the whole osd store
[19:42] <joshd> wonko_be: if you have the data replicated, you can wipe the osd and restart it
[19:42] <Tv> joshd: autotest going again
[19:42] <joshd> no way to recover the journal file itself, but the data will be safe
[19:42] <joshd> Tv: thanks
[19:43] <wonko_be> okay, thx for the info - a lost journal is as bad as losing the complete osd
[19:46] <gregaf> wonko_be: it's not necessarily that bad but it's safer to assume the worst, and right now I don't think our scrubbing code is up to detecting problems from journal issues
[19:47] <wonko_be> okay, no probs, now to figure out how to mkcephfs on only one osd :-)
[19:49] <gregaf> wonko_be: http://ceph.newdream.net/wiki/OSD_cluster_expansion/contraction
[19:49] <gregaf> that specifies how to add new nodes, but I think after wiping you'll just want to perform the osd-specific actions listed there :)
[19:49] <imcsk8> hello, i'm testing ceph and have a few questions: i've mounted the filesystem on my client machine did some tests: i've donwnloaded a iso file on the filesystem without problem and then i try a "ls -la" and it does not respond
[19:50] <wonko_be> gregaf: was looking there already
[19:50] <imcsk8> can i do something to avoid this and fine tune my filesystem?
[19:50] <wonko_be> and through the mkcephfs script, hoped there was something like mkcephfs --only osd0 or somehting like that
[19:51] <joshd> imcsk8: can you post the output of 'ceph -s'?
[19:51] <gregaf> wonko_be: hmm, mkcephfs on the osd in question might just work; you need to pass the -a arg for it to do stuff on other osds
[19:51] <gregaf> (sorry for the lack of specificity, I don't admin any clusters myself so I'm less familiar with setup)
[19:52] <imcsk8> joshd, http://pastebin.com/av184QFM
[19:52] <joshd> imcsk8: I suspect something went wrong with your cluster to cause a hang like that
[19:52] <imcsk8> johnl, i have to restart the ceph service in order for it to respond again
[19:52] <wonko_be> gregaf: still, without the -a it will rework both osd's that run on that node, while one is still good
[19:52] <wonko_be> but it is recovering alreay
[19:52] <wonko_be> thx
[19:52] <gregaf> ah, didn't realize you had multiple daemons
[19:52] <gregaf> np!
[19:53] <imcsk8> joshd, it hangs on one client and it works in other
[19:54] <joshd> imcsk8: is the other client doing lots of writes?
[19:55] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) has joined #ceph
[19:55] <imcsk8> joshd, in my first test it was, but right now none of the clients are writing
[19:56] <imcsk8> can't even unmount the filesystem on the client in question
[19:56] <imcsk8> oh wait, it just unmounted it
[19:56] <joshd> is this with the kernel client or cfuse?
[19:56] <imcsk8> kernel client
[19:57] <imcsk8> 2.6.35-22-generic #35-Ubuntu
[20:01] <joshd> it's strange that one of your osds was wrongly marked down - that means other osds couldn't communicate with it for some period of time
[20:02] <joshd> which version of ceph are you running? gregaf says we saw this behavior in past versions with incorrect capability passing
[20:04] <imcsk8> i have just one server running everything, ceph 0.25.2 on 2.6.38-gentoo
[20:06] <joshd> I'd suggest upgrading the client kernel - I'm not sure all the recent fixes are backported
[20:08] <imcsk8> joshd, ok, i'll check that thanks!!
[20:08] <joshd> the older versions were slow to release capabilities, so after accessing a directory from one client, another client would have to wait a while before it was able to read that directory
[20:09] <joshd> imcsk8: happy to help :)
[20:11] <imcsk8> joshd, i'm evaluating ceph for a filesystem cluster for the university of chihuahua
[20:12] <joshd> imcsk8: cool, let us know how it goes
[20:12] * sagelap (~sage@ has joined #ceph
[20:12] <imcsk8> joshd, yeah, i'll be around asking questions
[20:24] * neurodrone (~neurodron@cpe-76-180-162-12.buffalo.res.rr.com) has joined #ceph
[21:09] <wido> with this ceph.conf: http://zooi.widodh.nl/ceph/ceph.conf
[21:09] <wido> shouldn't logging to file be disabled?
[21:09] <wido> I do get logging to syslog, but my OSD's are still logging to file
[21:19] <joshd> wido: the log_to_file option was removed in bf1ae37425355f44741216a2449ae197e030cb8e, but the commit message says it should only log to a file if the 'log file' or 'log dir' options are set
[21:19] <joshd> cmccabe would know more
[21:21] <gregaf> I wonder if with the config changes they're now filled in by default?
[21:27] <wido> gregaf: I guess so, same goes for the pid. So when not setting a log file nor log dir, it would still log to a file
[21:59] * alexxy (~alexxy@ has joined #ceph
[22:03] * alexxy[home] (~alexxy@ Quit (Ping timeout: 480 seconds)
[22:05] <cmccabe> wido: the default for log_dir is /var/log/ceph
[22:05] <cmccabe> wido: so unless you add a line saying "log dir = ", it will create a log file there.
[22:07] * alexxy[home] (~alexxy@ has joined #ceph
[22:07] <cmccabe> wido: also, I don't think you want to set log per instance
[22:07] <cmccabe> wido: that activates our weird symlink dance
[22:08] <cmccabe> wido: so if you don't want a logfile, you certainly don't care about setting up the symlinks
[22:08] * alexxy (~alexxy@ Quit (Ping timeout: 480 seconds)
[22:08] * sagelap (~sage@ Quit (Read error: Operation timed out)
[22:09] * lxo (~aoliva@ Quit (Read error: Connection reset by peer)
[22:09] * lxo (~aoliva@ has joined #ceph
[22:09] <cmccabe> wido: so, long story short, add this to your conf to disable log files:
[22:09] <cmccabe> log file =
[22:09] <cmccabe> log dir =
[22:10] <cmccabe> gregaf: also, we've always activated log files by default. Nothing has changed there for a while.
[22:11] * s15y (~s15y@sac91-2-88-163-166-69.fbx.proxad.net) Quit (Quit: WeeChat 0.3.2)
[22:13] * s15y (~s15y@sac91-2-88-163-166-69.fbx.proxad.net) has joined #ceph
[22:14] * raso (~raso@debian-multimedia.org) has joined #ceph
[22:44] * verwilst (~verwilst@dD576FAAE.access.telenet.be) has joined #ceph
[22:57] * badari (~badari@ has joined #ceph
[22:57] <badari> i am new to ceph. i am having issues mounting ceph. can some one help ?
[22:58] <badari> I get following error when tried mounting
[22:58] <badari> mount error 5 = Input/output error
[22:59] <Tv> badari: that's the generic "could not talk to the monitor" error, might mean your monitor is not running, not at the ip you expect, etc
[23:00] <badari> I see - its connected to monitor (in the /var/log/messages)
[23:00] <badari> Apr 5 16:48:48 elm3b251 kernel: libceph: client4605 fsid bb3308e0-57e3-71ce-025e-065f939cc639
[23:00] <badari> Apr 5 16:48:48 elm3b251 kernel: libceph: mon0 session established
[23:03] <joshd> badari: do you have cephx turned on? is there any kind of error in /var/log/messages?
[23:03] <badari> joshd: sorry. don't know what is cephx turned on means :(
[23:04] <badari> joshd: i have a single machine.. using for mon, osd, msd - used sample .conf file and added few disks to it
[23:05] <joshd> cephx is an authorization system, which could be causing problems, but I think it's off by default
[23:05] <badari> joshd; how do I turn it on ?
[23:05] * alexxy[home] (~alexxy@ Quit (Ping timeout: 480 seconds)
[23:06] * sagelap (~sage@ has joined #ceph
[23:06] <joshd> badari: you don't need it on, but you add it to your ceph.conf as shown here: http://ceph.newdream.net/wiki/Cluster_configuration#Cephx_auth
[23:07] <badari> joshd: you are correct. I now removed "auth supported = cephx" and not used secretfile on the command line .. it worked
[23:08] <joshd> badari: cool
[23:09] * alexxy (~alexxy@ has joined #ceph
[23:12] * alexxy (~alexxy@ Quit (Remote host closed the connection)
[23:13] <badari> joshd: not so lucky.. kernel panics while doing writes to the filesystem :(
[23:13] * bchrisman (~Adium@c-98-207-207-62.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[23:14] <joshd> badari: which kernel version are you using? and which ceph version?
[23:14] <Tv> badari: you probably want to use the kernel from our ceph-client.git repo, it has bugfixes that are not yet upstream
[23:15] <badari> joshd, Tv: using 2.6.39-rc1, ceph-0.26
[23:16] * alexxy (~alexxy@ has joined #ceph
[23:16] <badari> Tv: I can try git kernel
[23:28] <wonko_be> gregaf: just to be complete (and I tested it), cosd -i X --mkjournal just does the trick
[23:28] <wonko_be> (on the correct node ofcourse)
[23:28] <gregaf> cool, glad you got it back up!
[23:28] <wonko_be> yeah, resetted the whole thing, got wondering, and broke it on purpose then :-)
[23:29] <wonko_be> and fixed it
[23:29] <wonko_be> cool, very nice stuff
[23:29] * allsystemsarego (~allsystem@ Quit (Quit: Leaving)
[23:33] <sagelap> badari, tv: we haven't tested -rc1. you might try v2.6.38 or the master branch in ceph-client.git (which is directly on top of v2.6.38). -rc1 has all the newly merged who knows what
[23:35] <badari> sagelap: sure. i will try git
[23:41] <sagelap> cmccabe: one thing i had to fix for v0.26 was the gtkmm version required in order to make the lenny package build. i just changed it from 2.13 to 2.12. did that version # come from anywhere specific? the changelog just says it fixed a build error
[23:41] <cmccabe> sagelap: I vaguely remember reading that number online somewhere
[23:41] <sagelap> ok. the packages build so i won't worry about it :)
[23:41] <cmccabe> sagelap: I have to admit it wasn't very scientific
[23:42] <josef> sagelap: stop releasing so often, you are killing me
[23:42] <cmccabe> sagelap: I just noticed someone blogged about how such-and-such software didn't work with gtkmm versions of 2.13 or less
[23:42] <cmccabe> sagelap: then noticed that I was getting the same error on centos 5
[23:42] <cmccabe> sagelap: so I assumed that ours also wouldn't work with 2.13 or less
[23:42] <sagelap> josef: :) i'm all for anything that makes rebuilding releases as trivial as possible!
[23:44] <cmccabe> sagelap: I'm pretty sure that centos 5's version was even earlier than 2.13
[23:44] <sagelap> tv: the other thing i did was revert the tcmalloc configure.ac patch. lenny apparently has the libgoogle-perftools-dev build-dep package, but it configure doesn't detect tcmalloc and errors out
[23:44] <cmccabe> sagelap: so the check should still work for them

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.