#ceph IRC Log


IRC Log for 2011-07-01

Timestamps are in GMT/BST.

[0:04] * monrad-51468 (~mmk@domitian.tdx.dk) Quit (Read error: Connection reset by peer)
[0:05] * monrad-51468 (~mmk@domitian.tdx.dk) has joined #ceph
[0:15] * jbd (~jbd@ks305592.kimsufi.com) has left #ceph
[0:24] * lxo (~aoliva@ has joined #ceph
[0:25] * aliguori (~anthony@ Quit (Quit: Ex-Chat)
[0:36] * lxo (~aoliva@ Quit (Ping timeout: 480 seconds)
[0:39] * lxo (~aoliva@ has joined #ceph
[0:53] * mtk (~mtk@ool-182c8e6c.dyn.optonline.net) Quit (Remote host closed the connection)
[0:55] * sung (~sung@doot.realfuckingnews.com) Quit (Quit: leaving)
[1:32] * greglap (~Adium@ has joined #ceph
[1:50] * dontHackMeBro (~dhmb@9KCAAAEAF.tor-irc.dnsbl.oftc.net) has joined #ceph
[1:56] * cmccabe (~cmccabe@c-24-23-254-199.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[2:20] * greglap (~Adium@ Quit (Read error: Connection reset by peer)
[2:23] * joshd (~joshd@ip-64-111-111-107.dreamhost.com) Quit (Quit: Leaving.)
[2:25] * Tv (~Tv|work@ip-64-111-111-107.dreamhost.com) Quit (Ping timeout: 480 seconds)
[3:06] * bchrisman (~Adium@70-35-37-146.static.wiline.com) Quit (Quit: Leaving.)
[3:56] * bchrisman (~Adium@c-98-207-207-62.hsd1.ca.comcast.net) has joined #ceph
[3:56] * yoshi (~yoshi@p15251-ipngn1601marunouchi.tokyo.ocn.ne.jp) has joined #ceph
[4:33] * dontHackMeBro (~dhmb@9KCAAAEAF.tor-irc.dnsbl.oftc.net) Quit (Ping timeout: 480 seconds)
[6:23] * greglap (~Adium@cpe-76-170-84-245.socal.res.rr.com) has joined #ceph
[8:18] * jbd (~jbd@ks305592.kimsufi.com) has joined #ceph
[8:50] * jbd (~jbd@ks305592.kimsufi.com) has left #ceph
[9:11] * lxo (~aoliva@ Quit (Ping timeout: 480 seconds)
[9:19] * lxo (~aoliva@ has joined #ceph
[10:15] * jbd (~jbd@ks305592.kimsufi.com) has joined #ceph
[11:05] * yoshi (~yoshi@p15251-ipngn1601marunouchi.tokyo.ocn.ne.jp) Quit (Remote host closed the connection)
[12:36] * morse (~morse@supercomputing.univpm.it) Quit (Remote host closed the connection)
[13:17] * morse (~morse@supercomputing.univpm.it) has joined #ceph
[13:30] * mtk (~mtk@ool-182c8e6c.dyn.optonline.net) has joined #ceph
[14:49] * lxo (~aoliva@ Quit (Ping timeout: 480 seconds)
[14:54] * aliguori (~anthony@cpe-70-123-132-139.austin.res.rr.com) has joined #ceph
[15:01] * DanielFriesen (~dantman@S0106001731dfdb56.vs.shawcable.net) Quit (Quit: http://daniel.friesen.name or ELSE!)
[15:28] * monrad-51468 (~mmk@domitian.tdx.dk) Quit (Quit: bla)
[15:28] * monrad (~mmk@domitian.tdx.dk) has joined #ceph
[15:35] * mtk (~mtk@ool-182c8e6c.dyn.optonline.net) Quit (Remote host closed the connection)
[15:38] * mtk (~mtk@ool-182c8e6c.dyn.optonline.net) has joined #ceph
[15:41] * mtk (~mtk@ool-182c8e6c.dyn.optonline.net) Quit ()
[15:42] * mtk (~mtk@ool-182c8e6c.dyn.optonline.net) has joined #ceph
[16:24] * greglap (~Adium@cpe-76-170-84-245.socal.res.rr.com) Quit (Read error: Operation timed out)
[16:39] * greglap (~Adium@cpe-76-170-84-245.socal.res.rr.com) has joined #ceph
[17:07] * aliguori (~anthony@cpe-70-123-132-139.austin.res.rr.com) Quit (Remote host closed the connection)
[17:29] * NashTrash (~Adium@ has joined #ceph
[17:29] <NashTrash> Hello Ceph'ers
[17:30] <NashTrash> I am new to ceph, and had a couple of questions.
[17:31] * aliguori (~anthony@ has joined #ceph
[17:31] <NashTrash> First off, as background, I am using cephx for auth support. If I run any commands from any machine other than the one I ran mkcephfs from, I get errors such as "monclient(hunting): MonClient::init(): Failed to create keyring"
[17:32] <NashTrash> This was after trying "rbd ls rbd"
[17:38] * Tv (~Tv|work@ip-64-111-111-107.dreamhost.com) has joined #ceph
[17:41] * bchrisman (~Adium@c-98-207-207-62.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[17:51] <greglap> NashTrash: hey, you're in a little early for us Californians :)
[17:51] <greglap> I suspect the error message there should be different, it probably can't find the keys it needs to authenticate with
[17:52] <NashTrash> Yeah. I copied the keyring.bin fine into /etc/ceph and it is now happy/
[17:53] <greglap> cool
[17:53] <NashTrash> It is morning in Cali, just a bit more morning that here in Nashville
[17:53] <NashTrash> I am trying to use Ceph as the volume backend for an Openstack cloud.
[17:54] <NashTrash> RBD via libvirt
[17:54] <greglap> cool
[18:03] * darktim (~andre@ticket1.nine.ch) Quit (Remote host closed the connection)
[18:10] <NashTrash> Thanks.
[18:10] * NashTrash (~Adium@ has left #ceph
[18:27] * bchrisman (~Adium@70-35-37-146.static.wiline.com) has joined #ceph
[18:27] * bchrisman (~Adium@70-35-37-146.static.wiline.com) Quit (Remote host closed the connection)
[18:32] * bchrisman (~Adium@70-35-37-146.static.wiline.com) has joined #ceph
[18:39] * cmccabe (~cmccabe@c-24-23-254-199.hsd1.ca.comcast.net) has joined #ceph
[18:47] * joshd (~joshd@ip-64-111-111-107.dreamhost.com) has joined #ceph
[18:53] * cmccabe1 (~cmccabe@c-24-23-254-199.hsd1.ca.comcast.net) has joined #ceph
[18:58] * cmccabe (~cmccabe@c-24-23-254-199.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[19:04] * greglap (~Adium@cpe-76-170-84-245.socal.res.rr.com) Quit (Quit: Leaving.)
[19:15] * greglap (~Adium@ has joined #ceph
[19:19] <sagewk> waiting a few min for yehuda
[19:25] <sagewk> ok lets do it now
[19:31] * sugoruyo (~george@athedsl-400474.home.otenet.gr) has joined #ceph
[19:31] <sugoruyo> hey folks, can anyone help me figure out why all of my MDSs are crashing?
[19:34] <greglap> sure, what's happening?
[19:38] <sugoruyo> well, basically i ran df on a directory with 13,000 files
[19:39] <sugoruyo> that hang, and then i saw mds0 was dead `ceph -s` reported it in replay(laggy or crashed)
[19:40] <sugoruyo> i restarted the whole thing and all 3 MDSs crashed (at the same state) so i restarted once more, now my MDSs are stuck in resolve, except mds0 which is in replay(laggy or crashed)
[19:40] <greglap> okay
[19:41] <greglap> so certainly there's something wrong with logical mds0's on-disk data
[19:41] <greglap> do you have logging and a backtrace?
[19:41] <sugoruyo> i haven't set any logging level in ceph.conf
[19:42] <greglap> okay, backtrace then?
[19:42] <sugoruyo> so it will be just what is logged by default, how do i get the backtrace?
[19:42] <greglap> the backtrace will show up in the default logging :)
[19:42] <sugoruyo> is that the output in dmesg?
[19:42] <sugoruyo> oh ok
[19:42] <sugoruyo> so mds.0.log should do? shall i put in pastie?
[19:43] <greglap> sure
[19:43] <sugoruyo> it's 2.3 megs - should i post a specific part?
[19:45] <greglap> just paste the backtrace for now
[19:47] <sugoruyo> http://www.pastie.org/2151066
[19:47] <sugoruyo> hope that's enough
[19:47] <greglap> yeah, that's what I was after
[19:48] <greglap> so, yeah, that's a bug that pops up when you run multi-mds systems right now
[19:48] <greglap> it's in the tracker but this sprint we've been focusing on other pieces
[19:49] <greglap> you should really admit defeat and run a single MDS at least until we say "multi-MDS should work, we hope" :)
[19:49] * NashTrash (~Adium@ has joined #ceph
[19:49] <NashTrash> Hello again Ceph'ers
[19:49] <sugoruyo> i see, i need to run multi-mds because i'm testing on machines with little RAM - 1gig P4s - and I've been having crashes just doing an ls on a dir
[19:49] <greglap> what kind of crashes?
[19:50] <greglap> 1GB isn't a lot, that's for sure, but I wouldn't expect it to crash
[19:50] <greglap> hi NashTrash
[19:50] <NashTrash> greglap: Hola
[19:50] <sugoruyo> not sure, similar symptoms to this though, just going into replay(laggy or crashed)
[19:50] <sugoruyo> is there a way for me to get unstuck?
[19:51] <greglap> hmm
[19:51] <greglap> I mean, you could always kill the journal, but that's sad
[19:51] <NashTrash> I am running iozone benchmark, and it has frozen. In the syslog I am getting an endless message "ceph: tid 29 timed out on osd7, will reset osd"
[19:51] <greglap> it might be possible to just kill the event causing this particular crash, but I'm not sure how that would work out, sagewk might be able to judge better
[19:52] <greglap> NashTrash: what are you running it on? (cfuse, kernel client, local fs on rbd, which version)
[19:52] <greglap> and what does ceph -s output?
[19:52] <sugoruyo> greglap: what's wrong with killing the journal? it crashed when i was doing a df on a big dir, so i should lose no data
[19:53] <NashTrash> I am using the kernel client I believe. And, ceph -s seems to show everything being OK.
[19:53] <greglap> sugoruyo: well it depends on how much data is in the journal; it might be that there's still stuff there
[19:53] <greglap> NashTrash: can you paste it?
[19:54] <sagewk> sugoruyo: can we get full mds logs of the replay? may be able to fix the bug
[19:54] <greglap> NashTrash: and which kernel version?
[19:54] <sugoruyo> well, since it's just a test system, i'd rather lose a few test data copies and just have to redo them, rather than mkcephfs again and have to copy the whole 90gigs currently in it
[19:54] <sagewk> that'll take some time sadly. or you can wipe out the mds journals and hope for the best :/
[19:54] <NashTrash> Here is the paste: http://pastebin.com/aYUFK3FP. Ubuntu 10.10 running 2.6.35-22-server.
[19:55] <greglap> sagewk: backtrace matches http://tracker.newdream.net/issues/1168
[19:55] <sugoruyo> sagewk: if you tell me exactly what you want, how i can get to it and how to get it to you
[19:55] <NashTrash> I had a bunch of debug output in syslog too on the client, but it mostly looks like iozone complaining about being blocked for more than 120 seconds.
[19:56] <sagewk> greglap: let's get the logs anyway.. could be many paths to that backtrace
[19:56] <greglap> sagewk: do you remember if iozone had any issues on the .35 kernel?
[19:56] <sagewk> that is pretty old, it probably did
[19:57] <greglap> does backports go back to that version or did the interfaces change too much after that?
[19:58] <NashTrash> kill does not seem to work on the iozone process. I can reboot the machine, but do I need to worry about anything on the ceph side of things?
[19:59] <greglap> NashTrash: not from an iozone run, worst that would happen is you lose some of the data that's still in the client's buffer and you have to wait through some timeouts before you do certain things to the files it was accessing
[20:00] <greglap> well, actually, that's the worst that would happen in any access pattern, but losing buffered data matters a lot more in some other circumstances ;)
[20:00] <greglap> unfortunately since Ceph is still under heavy development there's a lot of bugfixing that goes on in our current code that doesn't get backported
[20:01] <greglap> if you're in a position to upgrade your kernel version you probably want to do that; if not you might try testing with cfuse
[20:01] <NashTrash> Just in case I misunderstood you at any point, I am running ceph v0.30 but my client is 2.6.35.
[20:01] <greglap> it'll be much more up-to-date
[20:01] <greglap> yeah
[20:01] <NashTrash> cfuse. Got it
[20:02] <NashTrash> I am also using RBD, but that will be from the v0.30 version too.
[20:02] <sugoruyo> greglap, sagewk : what should i do about my logs? do you need them? how about getting unstuck?
[20:02] <greglap> cfuse doesn't support the ioctls for specifying data placement that the kernel does, but otherwise they've got feature parity and roughly equivalent speeds
[20:03] <greglap> NashTrash: the thing with Ceph is that being distributed it has servers and clients ??? when you're using v0.30 all your userspace tools can be up-to-date but your in-kernel clients are, well, from the kernel
[20:03] <greglap> which in this case is pretty old :(
[20:04] <sagewk> sugoruyo: add 'debug mds = 20' and 'debug ms = 1' to you [mds] section, and then restart the mds. once they crash attach the mds log files to a bug (probably need to bzip2 them first, they'll be big)
[20:04] <NashTrash> Yeah. I will just try to stick with client-side stuff that is from the same build generally.
[20:05] <greglap> NashTrash: I don't think you'll have any trouble from protocol incompatibilities; those don't change much and we do our best to preserve compatibility, it's just that part of the system state lives in the client so we can't fix client bugs without upgrading the client
[20:05] <greglap> like that issue you hit with iozone is almost certainly a bug in the client that we've since fixed in newer code
[20:06] <greglap> gotta run, be back in 20 folks
[20:06] <sugoruyo> sagewk: should i attach the logs from all MDSs, those that crash or the one that crashed originally? also is there a specific bug i should attach them to, or should i file one?
[20:07] <NashTrash> greglap: Thanks.
[20:07] * greglap (~Adium@ Quit (Read error: Connection reset by peer)
[20:13] * cmccabe1 (~cmccabe@c-24-23-254-199.hsd1.ca.comcast.net) has left #ceph
[20:33] <sugoruyo> sagewk: i have the logs from the MDSs, but they're 25-50 megs, what do i with them now?
[20:49] * aliguori (~anthony@ Quit (Quit: Ex-Chat)
[21:11] * aliguori (~anthony@ has joined #ceph
[21:18] <sugoruyo> so how can i get past this resolve stage on my MDSs and mount the system?
[21:29] * aliguori (~anthony@ Quit (Quit: Ex-Chat)
[22:00] * sugoruyo is now known as Guest606
[22:00] * sugoruyo (~george@athedsl-400474.home.otenet.gr) has joined #ceph
[22:00] * sugoruyo is now known as Guest607
[22:00] * sugoruyo (~george@athedsl-400474.home.otenet.gr) has joined #ceph
[22:01] * greglap (~Adium@ip-64-111-111-107.dreamhost.com) has joined #ceph
[22:02] <NashTrash> Hello. I am now trying to use cfuse to connect a client, but it still just hangs the client.
[22:02] <gregaf> sugoruyo: sorry, we had meetings and lunch, can you zip them up and post them in a new bug on the tracker?
[22:02] <NashTrash> Using v0.30 on all parts. I mount and it looks good, but if I try to ls the mount directory it just hangs forever
[22:02] <sugoruyo> they're 25-50 megs in bz2
[22:03] <gregaf> yeah, the tracker will take that now
[22:03] <sugoruyo> gregaf: i understand - it's just i'm in Greece and its 11pm here :P
[22:04] <gregaf> NashTrash: is this the same filesystem you were mounting before with the kernel client, or did you recreate it?
[22:04] <NashTrash> Do you mean using kfcephfs? I have not run that again.
[22:04] <gregaf> okay
[22:04] <gregaf> how are you mounting?
[22:04] <NashTrash> ceph -s shows everything happy. ceph -w shows lots of activity
[22:05] <gregaf> what kind of activity?
[22:05] <NashTrash> cfuse -d /mnt/ceph2
[22:05] <NashTrash> http://pastebin.com/dq0x1JAj for some of the ceph -w activity
[22:06] <gregaf> hmm, you don't need the -d in there, you might be hiding error messages by doing that
[22:06] <NashTrash> I am getting nothing anywhere on the client side.
[22:06] <gregaf> ah, yep, that activity is the OSDs scrubbing and making sure their data is okay (it's very primitive at the moment)
[22:06] <NashTrash> I tried it without the -d and it still hung and put nothing into the logs
[22:06] <gregaf> it did mount, though?
[22:07] <NashTrash> Here is the mounting: http://pastebin.com/CRx6DMwK
[22:07] * Guest606 (~george@athedsl-400474.home.otenet.gr) Quit (Ping timeout: 480 seconds)
[22:07] <gregaf> okay, try
[22:08] <gregaf> cfuse /mnt/ceph2 ???debug_client 20 ???debug_ms 1 ???log-file=/path/to/good/location
[22:08] * Guest607 (~george@athedsl-400474.home.otenet.gr) Quit (Ping timeout: 480 seconds)
[22:08] <gregaf> and pastebin that log
[22:08] <NashTrash> Ok. I have to go manually reboot the machine.
[22:09] <NashTrash> brb
[22:09] <sugoruyo> gregaf: is there any way i can get that f/s with the hanging/crashing MDSs to mount? i really need to get a couple of files off it
[22:10] <gregaf> sugoruyo: not unless you want to wipe out your journals again
[22:10] <sugoruyo> gregaf: the MDS journals?
[22:10] <gregaf> if you can wait I will talk to Sage about it and see if there's something else we can do or if we can solve that bug quickly
[22:10] <gregaf> yeah
[22:10] <gregaf> but that is risky as always
[22:12] <sugoruyo> well i really need to resume testing within the next 24 hours, so a mkcephfs might be in order anyway - given that i'll lose everything by doing that i figure i've nothing to lose by trying to wipe out the journals and trying to get to my files before giving up on them
[22:12] <sugoruyo> but i don't know how to wipe out the journal
[22:12] <gregaf> cmds ???reset-journal, I think
[22:12] <gregaf> cmds will tell you if you run it without arguments or with invalid ones
[22:13] <gregaf> you really shouldn't be storing important files in a known-unstable config, though :(
[22:13] <gregaf> I promise we'll tell you when we think it will work
[22:13] <sugoruyo> gregaf: well i was actually storing copies
[22:14] <sugoruyo> but then i changed the files (they're python code to test some things with ceph)
[22:14] <sugoruyo> and ran them locally, i just wanna get to those changes so i don't have to rewrite them
[22:14] <gregaf> heh
[22:15] <sugoruyo> i'm resetting the journal - fingers crossed
[22:15] <NashTrash> gregaf: I tried the mount line you gave me and it died with "fuse_parse_cmdline failed" on argument -debug_client
[22:15] <NashTrash> says its invalid
[22:15] <gregaf> oh, that's two dashes, not one
[22:15] <NashTrash> ah
[22:15] <gregaf> sorry, I think my client merged them into an emdash
[22:15] <NashTrash> same on debug_ms and log-file?
[22:15] <gregaf> yeah
[22:16] <NashTrash> fuse: invalid argument `???-debug_client'
[22:16] <gregaf> well that looks to me like now you've got an emdash and a hyphen
[22:16] <gregaf> ?
[22:17] <NashTrash> it is two hypens on my terminal
[22:17] <NashTrash> darn Adium
[22:17] <NashTrash> Nope, it was an emdash
[22:17] <NashTrash> Now running
[22:17] <gregaf> heh :p
[22:17] <NashTrash> What should I try? ls the directory?
[22:18] <gregaf> can you just pastebin the log file now?
[22:18] <gregaf> once you've done that do then sure
[22:18] <NashTrash> http://pastebin.com/fhQb5U8X
[22:19] <gregaf> okay, that does look right at first glance
[22:20] <gregaf> do the ls and give me the extra logging from that too :)
[22:20] <NashTrash> ls -all /mnt/ceph2 froze the one terminal window. Logging is still going
[22:21] <NashTrash> http://pastebin.com/Jr8CpjkY
[22:21] * nolan (~nolan@phong.sigbus.net) Quit (Ping timeout: 480 seconds)
[22:24] <NashTrash> Just in case, here is my ceph.conf: http://pastebin.com/j8jzTjBJ
[22:25] <gregaf> your ceph.conf looks fine, although having 2 monitors instead of just one isn't helping you any ??? the cluster requires a strict majority in order to progress
[22:25] <gregaf> the logging you've given me looks okay as far as it goes, although there is a request in to the MDS that doesn't get answered ??? I'm not sure if that's because it won't get answered or because you cut off the logs too early :)
[22:26] <gregaf> can you grep the log for "handle_client_reply" and show me what you get?
[22:31] <NashTrash> 2011-07-01 15:17:36.002045 7fdba0b9c700 client4762 handle_client_reply got a reply. Safe:1 tid 1
[22:31] <NashTrash> 2011-07-01 15:17:36.002216 7fdba0b9c700 client4762 handle_client_reply signalling caller 0x7fff3fbc8a40
[22:31] <NashTrash> 2011-07-01 15:17:36.002226 7fdba0b9c700 client4762 handle_client_reply awaiting kickback on tid 1 0x7fdba0b9b600
[22:31] <NashTrash> 2011-07-01 15:20:06.376031 7fdba0b9c700 client4762 handle_client_reply got a reply. Safe:1 tid 2
[22:31] <NashTrash> 2011-07-01 15:20:06.376250 7fdba0b9c700 client4762 handle_client_reply signalling caller 0x7fff3fbc8610
[22:31] <NashTrash> 2011-07-01 15:20:06.376260 7fdba0b9c700 client4762 handle_client_reply awaiting kickback on tid 2 0x7fdba0b9b600
[22:31] <NashTrash> The "ls" command is still hung
[22:32] <gregaf> yeah
[22:32] <gregaf> okay, so the client has a request out to the MDS that isn't getting answered
[22:32] <NashTrash> yeap
[22:32] <NashTrash> How do I safely shutdown cfuse?
[22:32] <NashTrash> umount?
[22:33] <NashTrash> Ha
[22:33] <gregaf> if it's hanging you'll just have to kill it
[22:33] <gregaf> kill ???9 cfuse_pid; fusermount -u mntpoint
[22:33] <gregaf> drat, no MDS logging in your conf
[22:33] <NashTrash> kill ???9 cfuse_pid; fusermount -u /mnt/ceph2
[22:33] <NashTrash> will do
[22:33] <gregaf> yeah
[22:34] <NashTrash> fusermount: failed to unmount /mnt/ceph2: Device or resource busy
[22:34] <gregaf> oh, yeah, you'll need to move your terminal cwd out of it after killing cfuse
[22:34] * cmccabe1 (~cmccabe@c-24-23-254-199.hsd1.ca.comcast.net) has joined #ceph
[22:35] <gregaf> if you'd had logging on for your MDS I could debug it, but without logs I just have to guess that something I haven't thought of went wrong when you were using the kernel client :/
[22:35] <NashTrash> Ok. How do I turn on logging?
[22:35] <NashTrash> I am happy to recreate the ceph fs too. Nothing in it.
[22:35] <gregaf> if you want to be super-helpful :)
[22:36] <gregaf> you can add "debug ms = 1" and "debug mds = 20" to your MDS's ceph.conf and restart it
[22:36] <NashTrash> Happy to help. We are doing a bake off of Ceph vs. a bunch of other filesystems and I want Ceph to win
[22:36] <gregaf> and then try to mount and ls again
[22:37] <gregaf> if it was some bizarre ephemeral problem (probably from something that went wrong with the kernel client) then it'll just disappear and things will work, if something actually went bad on-disk then it'll show up here and we can debug it
[22:37] <gregaf> what other FSes are you looking at?
[22:37] <NashTrash> moose, gluster, isilon
[22:38] <gregaf> wow, one of those is not like the other
[22:38] <NashTrash> You thiink
[22:39] <gregaf> well, let's see: ceph, moose, gluster, isilon
[22:39] <gregaf> OSS, OSS, OSS, proprietary company ;)
[22:39] <NashTrash> Hmmm???..
[22:39] <gregaf> it's just unusual for companies to be looking at both at once
[22:40] <gregaf> in my limited experience
[22:40] <NashTrash> We are a research lab.
[22:40] <NashTrash> Rather spend the bucks on more hardware, but need to make sure it will work for our experiments
[22:40] <gregaf> ah
[22:40] <NashTrash> service ceph -a stop ->>
[22:40] <NashTrash> === osd.7 ===
[22:40] <NashTrash> Stopping Ceph osd.7 on cloudblock3...kill 3422...done
[22:40] <NashTrash> Unmounting Btrfs on cloudblock3:/mnt/btrfs/osd7
[22:40] <NashTrash> umount: /mnt/btrfs/osd7: device is busy.
[22:40] <NashTrash> (In some cases useful info about processes that use
[22:40] <NashTrash> the device is found by lsof(8) or fuser(1))
[22:41] <NashTrash> Not letting me shut it down
[22:41] <bchrisman> think there are some nat'l labs working with/on ceph stuff aren't there?
[22:41] <gregaf> uh, joshd?
[22:41] <gregaf> or sjust?
[22:41] <gregaf> (these guys are a lot more familiar with the OSD btrfs interactions and likely problems)
[22:42] <gregaf> bchrisman: Sandia has one guy on it half-time or something, I think; and some of the others are playing with it as hobbies or as a target for other research or something, I don't think anybody's doing anything too serious quite yet though
[22:42] <joshd> I think you just need to wait for btrfs to finish committing before you can unmount it
[22:44] <bchrisman> considering the labs' romance with LUSTRE, that's a lot I think? :)
[22:44] <joshd> also, since you're running 2.6.35, you may run into bugs in btrfs (there have been plenty fixed since then)
[22:46] <NashTrash> There are indeed. We are a university (Vanderbilt) worked on a DARPA funded research project.
[22:46] <NashTrash> Still not unmounting. How long can btrfs take?
[22:47] <sjust> NashTrash: actually, quite a while
[22:47] <NashTrash> Ha
[22:47] <sjust> does dmesg mention anything?
[22:47] <NashTrash> Ok. Is there anyway to get a more up to date btrfs without having to update the kernel (our sys admins like to stick with vanilla 10.10 for now)
[22:47] <sjust> ah
[22:48] <gregaf> everything should work fine on ext4 too (and I think ext3, although I forget)
[22:48] <NashTrash> Lots of btrfs trace info in dmesg
[22:48] <gregaf> as long as you don't want to use more than 4k per inode of xattrs
[22:49] <NashTrash> [75233.008784] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[22:49] <NashTrash> [75233.008832] cosd D 000000010070a29f 0 3454 1 0x00000000
[22:49] <NashTrash> [75233.008837] ffff88012bcc7b28 0000000000000082 ffff880100000000 0000000000015b40
[22:49] <NashTrash> [75233.008842] ffff88012bcc7fd8 0000000000015b40 ffff88012bcc7fd8 ffff8801284edbc0
[22:49] <NashTrash> [75233.008847] 0000000000015b40 0000000000015b40 ffff88012bcc7fd8 0000000000015b40
[22:49] <NashTrash> [75233.008851] Call Trace:
[22:49] <NashTrash> [75233.008872] [<ffffffffa025106d>] btrfs_start_ordered_extent+0x6d/0xc0 [btrfs]
[22:49] <NashTrash> [75233.008880] [<ffffffff8107fb90>] ? autoremove_wake_function+0x0/0x40
[22:49] <NashTrash> [75233.008894] [<ffffffffa0251859>] btrfs_wait_ordered_range+0xb9/0x160 [btrfs]
[22:49] <NashTrash> [75233.008909] [<ffffffffa02443f2>] btrfs_file_aio_write+0x592/0x9c0 [btrfs]
[22:49] <NashTrash> [75233.008923] [<ffffffffa0235e64>] ? __btrfs_end_transaction+0x144/0x220 [btrfs]
[22:49] <NashTrash> [75233.008938] [<ffffffffa0243e60>] ? btrfs_file_aio_write+0x0/0x9c0 [btrfs]
[22:49] <NashTrash> [75233.008944] [<ffffffff81154443>] do_sync_readv_writev+0xd3/0x110
[22:49] <NashTrash> [75233.008948] [<ffffffff8115455a>] ? do_sync_write+0xda/0x120
[22:49] <NashTrash> [75233.008954] [<ffffffff81290df8>] ? apparmor_file_permission+0x18/0x20
[22:49] <NashTrash> [75233.008961] [<ffffffff81260396>] ? security_file_permission+0x16/0x20
[22:49] <NashTrash> [75233.008965] [<ffffffff8115547f>] do_readv_writev+0xcf/0x1f0
[22:49] <NashTrash> [75233.008969] [<ffffffff811555e8>] vfs_writev+0x48/0x60
[22:49] <NashTrash> [75233.008972] [<ffffffff81155711>] sys_writev+0x51/0xb0
[22:49] <NashTrash> [75233.008978] [<ffffffff8100a0f2>] system_call_fastpath+0x16/0x1b
[22:49] <NashTrash> ext4 would be fine. I assume I just have to fdisk the drives and format them myself with xattrs, right?
[22:49] <NashTrash> and not use ???mkbtrfs when I mkcephfs
[22:49] <sjust> NashTrash: yeah, that should do it
[22:50] <NashTrash> Ok. That sounds like a job for Tuesday. Will v0.31 be out by then?
[22:51] <sugoruyo> bye folks, thanks for the help
[22:51] * sugoruyo (~george@athedsl-400474.home.otenet.gr) Quit (Quit: sugoruyo)
[22:52] <NashTrash> I am heading out too. Thanks for your help. I will give an update on Tuesday.
[22:52] <gregaf> NashTrash: I think .31 will be another week ??? we're finishing the sprint now and will leave it in a branch for a week for a little more testing and in case we find new bugs
[22:53] <NashTrash> Sounds good. I am really excited to be working with it. If we can show it works for us, we will be bringing up about 500TB pretty quickly.
[22:54] * NashTrash (~Adium@ has left #ceph
[22:54] <gregaf> if you want to write up your workload for us it might be good to let us see what features you'll need to be stable :)
[22:54] <gregaf> different pieces are at very different stages of testing and maturity

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.