#ceph IRC Log


IRC Log for 2010-10-11

Timestamps are in GMT/BST.

[0:24] * DJCapelis (~djc@capelis.dj) Quit (Remote host closed the connection)
[0:24] * conner (~conner@leo.tuc.noao.edu) Quit (Ping timeout: 480 seconds)
[0:30] * DJCapelis (~djc@capelis.dj) has joined #ceph
[1:45] * DJCapelis (~djc@capelis.dj) Quit (Remote host closed the connection)
[1:58] * DJCapelis (~djc@capelis.dj) has joined #ceph
[2:27] * tjikkun (~tjikkun@2001:7b8:356:0:204:bff:fe80:8080) Quit (Ping timeout: 480 seconds)
[2:28] * tjikkun (~tjikkun@2001:7b8:356:0:204:bff:fe80:8080) has joined #ceph
[2:36] * tjikkun (~tjikkun@2001:7b8:356:0:204:bff:fe80:8080) Quit (Ping timeout: 480 seconds)
[2:37] * tjikkun (~tjikkun@2001:7b8:356:0:204:bff:fe80:8080) has joined #ceph
[3:00] * tjikkun (~tjikkun@2001:7b8:356:0:204:bff:fe80:8080) Quit (Ping timeout: 480 seconds)
[3:00] * tjikkun (~tjikkun@2001:7b8:356:0:204:bff:fe80:8080) has joined #ceph
[3:09] * tjikkun (~tjikkun@2001:7b8:356:0:204:bff:fe80:8080) Quit (Ping timeout: 480 seconds)
[3:10] * tjikkun (~tjikkun@195-240-122-237.ip.telfort.nl) has joined #ceph
[5:54] * tjikkun (~tjikkun@195-240-122-237.ip.telfort.nl) Quit (Ping timeout: 480 seconds)
[5:58] * sockmister (82d8d198@ircip1.mibbit.com) has joined #ceph
[5:59] <sockmister> erm; anyone alive ? :p
[5:59] <cowbar> yep!
[5:59] <cowbar> at least, have a pulse last I checked.
[6:00] <sockmister> argh great,
[6:00] <sockmister> haven't used irc for really... long time; heh
[6:02] <sockmister> so anyone got the latest, k-client compiled and be able to mount? I can't seem to compile it for master-backport
[6:03] <cowbar> now you're into realm of things I can't help with. :)
[6:03] <cowbar> I juust lurk.
[6:03] <cowbar> know little about ceph.
[6:03] <cowbar> but if you stick around for like 12 hours, the guys who know will be online
[6:03] <sockmister> hehe;
[6:04] <sockmister> in case anyone, I've already log here, @ http://article.gmane.org/gmane.comp.file-systems.ceph.devel/1118
[6:05] <sockmister> thx >.<
[6:08] <sockmister> Sage justed answered, wow he is fast :)
[6:10] <sage> sockmister: oh hi.. are you retrying the master-backport now?
[6:10] <sockmister> yes, i am trying it now, give me a minute :)
[6:15] <sockmister> Sage, now I get "WARNING: "task_dirty_inc" [/home/aaa/ceph/ceph-client-standalone/ceph.ko] undefined!"
[6:15] <sage> this is on
[6:15] <sage> ?
[6:15] <sockmister> yes
[6:15] <sockmister> Linux ss1 2.6.35-02063507-generic #201009290909 SMP Wed Sep 29 09:11:05 UTC 2010 x86_64 GNU/Linux
[6:16] <sage> * master-backport 0184d86 fix account_page_dirtied (2.6.36, not 35)
[6:16] <sage> on that commit in -standalone.git?
[6:16] <sockmister> argh;;
[6:17] <sockmister> so no for 35.7 ?
[6:18] <sage> it just built for me on
[6:18] <sockmister> also, I keep getting "failed to parse ceph_options" whenever I tried to mount, how can I do anything about it?
[6:18] <sage> were you using the unstable branch?
[6:19] <sockmister> yes I used unstable branch on 2.6.32-24 on other PC, and it can mount (but not really stable, when I run dbench)
[6:20] <sage> the current unstable is stale/buggy.. need to fix a few things before it syncs up with the main ceph-client.git
[6:20] <sockmister> i see, what knerl version and client do you recommend to use?
[6:20] <sage> (the new code is broken into multiple modules and the -standalone scripts aren't updated yet)
[6:21] <sage> ideally, 2.6.36 (almost done). or master-backport and anything relatively recent.
[6:22] <sockmister> 2.6.36-rc7 ?
[6:24] <sage> sure. or just stick with and master-backport, that's probably easiest.
[6:25] <sockmister> thx, I'm just downloading 2.6.36-rc7 (the latest), will try this, but
[6:26] <sockmister> but importantly, something got broke, and no matter what I do , the mount keep sailing
[6:26] <sockmister> *failing*
[6:26] <sockmister> "failed to parse ceph_options"
[6:26] <sage> wait.. even with master-backport?
[6:27] <sockmister> yes, -- I don't think this is to do with branch, it even give me the error when I did rmmod ceph
[6:28] <sockmister> on 2.35.7
[6:29] <sage> hmm, not sure where that is coming from. where do you see it?
[6:30] <sockmister> right after I do mount -t ceph /ceph
[6:31] <sockmister> it returns almost instant, without any delay
[6:32] <sockmister> dmesg doesnt show any, anyhow to get rid of entire kclient, possible conflicts in the system?
[6:32] <sockmister> or even what kclient i have compiled..
[6:32] <sage> yeah. that's weird. i can't find "ceph_options" anywhere in user or kernel git either.
[6:34] <sage> can you paste relevant terminal output to http://tracker.newdream.net/issues/475?
[6:34] <sage> maybe 'strace mount -t ceph /ceph' and see if that has any clues?
[6:37] <sockmister> it is coming from sbin, and I've tried the mount.ceph, too, I just did strace, bunch of outputs, where should I paste to?
[6:37] <sage> attach a file to that bug?
[6:38] <sockmister> oki!
[6:38] <sockmister> do I have the access, sorry can't seem to find a way to respond to that link
[6:40] <sage> oh, you probably need to create a user in the tracker.
[6:42] <sockmister> thx, will do
[7:00] * xilei (~xilei@ has joined #ceph
[7:09] * sockmister (82d8d198@ircip1.mibbit.com) Quit (Quit: http://www.mibbit.com ajax IRC Client)
[7:12] * DJL (82d8d198@ircip1.mibbit.com) has joined #ceph
[7:33] * DJL (82d8d198@ircip1.mibbit.com) Quit (Quit: http://www.mibbit.com ajax IRC Client)
[7:37] * SwiftLayer (~sm@c-24-19-82-48.hsd1.wa.comcast.net) has joined #ceph
[7:37] <SwiftLayer> hello
[7:37] <SwiftLayer> I know Ceph is no where near completion, but I had some questions, will it be loadable as a block device? maybe support LVMoIP?
[8:06] * tjikkun (~tjikkun@2001:7b8:356:0:204:bff:fe80:8080) has joined #ceph
[9:26] * gregorg_taf (~Greg@ Quit (Quit: Quitte)
[9:27] * gregorg (~Greg@ has joined #ceph
[9:31] * allsystemsarego (~allsystem@ has joined #ceph
[9:35] * tjikkun (~tjikkun@2001:7b8:356:0:204:bff:fe80:8080) Quit (Ping timeout: 480 seconds)
[9:35] * tjikkun (~tjikkun@195-240-122-237.ip.telfort.nl) has joined #ceph
[10:01] <wido> SwiftLayer: yes, you can use RBD, the Rados Block Device
[10:01] <wido> check out the Wiki page
[10:02] <wido> SwiftLayer: http://ceph.newdream.net/wiki/Rbd
[10:02] * Yoric (~David@ has joined #ceph
[10:05] * Yoric (~David@ Quit (Read error: Connection reset by peer)
[10:05] * Yoric (~David@ has joined #ceph
[10:08] * Yoric_ (~David@ has joined #ceph
[10:08] * Yoric (~David@ Quit (Read error: Connection reset by peer)
[10:08] * Yoric_ is now known as Yoric
[10:09] * Yoric (~David@ Quit (Read error: Connection reset by peer)
[10:43] * Yoric (~David@ has joined #ceph
[11:46] * gregaf (~Adium@ip-66-33-206-8.dreamhost.com) Quit (Ping timeout: 480 seconds)
[11:46] * yehudasa (~yehudasa@ip-66-33-206-8.dreamhost.com) Quit (Ping timeout: 480 seconds)
[11:46] * sagewk (~sage@ip-66-33-206-8.dreamhost.com) Quit (Ping timeout: 480 seconds)
[12:34] * xilei (~xilei@ Quit (Quit: Leaving)
[14:49] * Yoric (~David@ Quit (Quit: Yoric)
[15:53] * Yoric (~David@ has joined #ceph
[16:10] * Yoric (~David@ Quit (Read error: Connection reset by peer)
[16:10] * Yoric (~David@ has joined #ceph
[16:53] * yehudasa (~yehudasa@ip-66-33-206-8.dreamhost.com) has joined #ceph
[16:53] * greglap (~Adium@ has joined #ceph
[16:57] * gregaf (~Adium@ip-66-33-206-8.dreamhost.com) has joined #ceph
[17:42] * greglap (~Adium@ Quit (Ping timeout: 480 seconds)
[17:52] * greglap (~Adium@ip-66-33-206-8.dreamhost.com) has joined #ceph
[17:53] * greglap (~Adium@ip-66-33-206-8.dreamhost.com) Quit ()
[18:14] <wido> hi
[18:15] <wido> since two weeks i'm having some strange issues, a "ceph -s" takes a long time
[18:15] <wido> $ time ceph -s
[18:15] <wido> real0m9.261s
[18:15] <wido> i've checked the connectivity to all the monitors, but that seems to be fine
[18:17] <wido> i've set "debug ms" to 20, that gave me "reader reading tag..." for a few times, before continuing
[18:21] <gregaf> what do you mean a few times?
[18:22] <gregaf> that's the initial message for every incoming pipe (ie, session)
[18:22] * sagewk (~sage@ip-66-33-206-8.dreamhost.com) has joined #ceph
[18:22] * iggy (~iggy@theiggy.com) Quit (Quit: leaving)
[18:23] <wido> oh, ok
[18:23] <wido> well, it comes back a few times, but after a few seconds I get the output
[18:24] <wido> it seems to switch to another monitor before completing, but all the monitors are up and running
[18:24] <alexxy> hi all
[18:24] <alexxy> how can i add mds to running cluster?
[18:32] <gregaf> alexxy: you'll need to add the mds to your config file (at least on your monitors), then "ceph mds set_max_mds n" (where n is your current number of MDSes — it's zero-indexed), and then start up your new mds
[18:33] <gregaf> Sage just added a bug to the tracker for a wiki page, too, so hopefully we'll have it documented before the next person needs it!
[18:33] <alexxy> =)
[18:33] <alexxy> thanks i'll try it
[18:34] <gregaf> err, sorry, n+1, not n
[18:35] <gregaf> the checks aren't set up quite how I thought they were
[18:35] * alexxy created 6node test cluster setup
[18:36] <alexxy> btw is there any plans to add rdma devices support
[18:36] <alexxy> e.g. infiniband?
[18:36] <sagewk> alexxy: general plans, yes. no specific plan as of yet. there is lots of interest
[18:37] <alexxy> i can give accsess to ib hw if needed
[18:39] <sagewk> alexxy: thanks, will keep that in mind. what kind of hardware?
[18:54] <sagewk> interesting reading on CAP: http://codahale.com/you-cant-sacrifice-partition-tolerance/
[18:59] * Yoric (~David@ Quit (Quit: Yoric)
[19:26] <wido> sagewk: cool :) I've also checked out Sheepdog, and network splitting is one of the things I always worry about
[19:26] <wido> with Ceph it's all about strategic monitor placement, right?
[19:30] <wido> gregaf: about the monitors, all my 3 mon's are online, but two don't seem to respond to my queries
[19:30] <wido> ceph -s show 3 monitors up, but when forcing the monitor with: ceph -m [ip] -s I get timeouts
[19:48] <wido> Another question: A MDS has the caps "[osd] allow *" isn't that a bit to wide? Shouldn't it have "rx" on "data" and "rwx" on "metadata" ?
[19:52] <alexxy> sagewk: two nodes with mthca adapters
[19:52] <alexxy> =)
[19:52] <alexxy> for example
[19:52] <alexxy> may be other systems
[20:37] <alexxy> btw
[20:37] <alexxy> is it possible to add journal on osd's?
[20:37] <alexxy> how large it should be
[20:37] <alexxy> i has 512G osd's
[20:38] <darkfader> alexxy: i had asked about ib support and promised to throw in a hca or two as motivation
[20:38] <darkfader> maybe you wanna join
[20:38] <alexxy> well i can give remote access to hw
[20:38] <alexxy> with ipmi =)
[20:42] <darkfader> counting you there's already 3 people here with infiniband
[20:42] <darkfader> not too bad
[20:45] <alexxy> =)
[21:03] <wido> alexxy: Yes, you can add a journal
[21:04] <wido> but it should just be enough for a few seconds, around 2GB of journal size should be enough
[21:08] <alexxy> ok
[21:08] * alexxy ready to commit ceph to gentoo main tree =)
[21:12] <alexxy> wido: what should i do to add journal?
[21:21] <gregaf> wido: we're in meetings initiating new people all day, I'll talk to you about your monitors later or tomorrow
[21:22] <gregaf> alexxy: wido: an OSD journal of 2GB is actually pretty large, although it won't break anything at that size
[21:23] <gregaf> wido: oh, and that wiki looks pretty good, but I was actually wrong about what "N" is if you look back up there (checks aren't quite what I thought), we'll need to fix that before we mark the ticket resolved :)
[21:35] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) has joined #ceph
[21:38] <wido> alexxy: you should run the cosd command manually with: cosd -i <osdnum> -c /etc/ceph/ceph.conf --mkjournal
[21:39] <wido> gregaf: no problem at all! As long as they are new Ceph developers ;)
[21:39] <wido> i'm not sure what my mon problem is, you could log on to logger.ceph.widodh.nl (your key (macbook)) is still in the auth list. You will see that mon.node13/14 aren't responding, but are up and running
[21:40] <wido> i'll go afk shortly too
[21:42] <wido> i've got some more issues, but that's something for tomorrow then :)
[21:45] * MarkN (~nathan@ Quit (Ping timeout: 480 seconds)
[21:48] * MarkN (~nathan@ has joined #ceph
[21:49] <alexxy> hmm
[21:49] <alexxy> sees i have strance situation with monitors
[21:49] <alexxy> 10.10.11_23:49:03.137612 log 10.10.11_23:49:02.570417 mon1 3038 : [WRN]
[21:49] <alexxy> 10.10.11_23:49:04.213380 log 10.10.11_23:49:02.984049 mon1 3039 : [WRN] lease_expire from mon0 was sent from future time 10.10.11_23:49:03.076538, clocks not synchronized
[21:49] <darkfader> alexxy: is your ntp setup all fine?
[21:50] <alexxy> yep
[21:50] <darkfader> then you might need some special option
[21:50] <darkfader> i think it was called mon_wiggle_room = 1 or something around that
[21:50] <darkfader> just wait for someone that knows.
[21:50] <darkfader> :)
[21:51] <alexxy> http://paste.pocoo.org/show/274163/
[21:52] <alexxy> where should this option be?
[21:53] <darkfader> i can't help you, i didnt use my ceph vms for 2 or 3 months
[21:53] <darkfader> sorry :(
[21:53] <darkfader> just trying to get it all in shape again so i can continue where i'd left off
[21:53] <wido> alexxy: mon clock drift allowed = 1
[21:53] <wido> mon clock drift warn backoff = 30
[21:54] <wido> place it in the [mon] section
[21:54] <wido> but even better, make sure the clocks are running in sync!
[21:55] <wido> Ceph is pretty strict about time
[21:56] <wido> when you are syncing from a NTP server over the net, the variable latency could even be a problem
[21:56] <wido> best is to place a NTP server in your local network, let that one sync with the internet
[21:56] <wido> and both your mon's sync with that NTP server
[21:56] <wido> i'm afk, ttyl!
[21:56] <alexxy> still have this issue
[21:59] <darkfader> well just searched based on what wido said.
[21:59] <darkfader> the option is supposed to work unless your clocks ARE apart
[22:06] <alexxy> hmmm
[22:11] <darkfader> are there any centos fuse client rpms yet?
[22:20] <darkfader> the dpkg scripts really need to stop starting ceph
[22:20] <darkfader> this is the worst nightmare i can imagine.
[22:21] <darkfader> tried to install nfs-client. it decides this is a good time to start ceph a second time
[22:21] <darkfader> ceph says that won't work, so nfs install fails
[22:21] <alexxy> 10.10.12_00:21:18.914813 log 10.10.12_00:21:17.874208 mon0 794 : [WRN] lease_ack from mon1 was sent from future time 10.10.12_00:21:17.887758, clocks not synchronized.
[22:22] <gregaf> alexxy: that means your monitor clocks aren't quite synced up
[22:22] <gregaf> although based on the timestamp it shouldn't be significant
[22:22] <gregaf> what version of Ceph are you running, and what options did you end up passing in?
[22:22] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) Quit (Quit: Yoric)
[22:23] <alexxy> its 0.21.3
[22:24] <alexxy> http://paste.pocoo.org/show/274178/
[22:24] <alexxy> ceph.conf ^^^
[22:26] <gregaf> ah, that version calls the option "mon lease wiggle room" instead of "mon clock drift allowed"
[22:26] <alexxy> 10.10.12_00:25:50.597298 7f0bacaf2710 -- >> pipe(0x7f0ba8001510 sd=18 pgs=0 cs=0 l=0).connect claims to be not - wrong node!
[22:26] <alexxy> also i get this with mds
[22:27] <gregaf> see http://ceph.newdream.net/wiki/Ceph.conf for more on the options :)
[22:28] <gregaf> umm, I think the "wrong node" message is usually related to a daemon restart that hasn't propagated through the system yet
[22:28] <gregaf> gotta go now, though!
[23:15] * allsystemsarego (~allsystem@ Quit (Quit: Leaving)

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.