#ceph IRC Log


IRC Log for 2010-12-07

Timestamps are in GMT/BST.

[2:32] * sjust (~sam@ip-66-33-206-8.dreamhost.com) Quit (Ping timeout: 480 seconds)
[2:47] * ghaskins_mobile (~ghaskins_@66-189-113-47.dhcp.oxfr.ma.charter.com) has joined #ceph
[2:49] * joshd (~joshd@ip-66-33-206-8.dreamhost.com) Quit (Quit: Leaving.)
[2:55] * ghaskins_mobile (~ghaskins_@66-189-113-47.dhcp.oxfr.ma.charter.com) Quit (Quit: This computer has gone to sleep)
[3:18] * bchrisman (~Adium@70-35-37-146.static.wiline.com) Quit (Quit: Leaving.)
[5:23] * bchrisman (~Adium@c-24-130-226-22.hsd1.ca.comcast.net) has joined #ceph
[5:45] <bchrisman> I've started up daemons.. and I'm seeing a lot of 'fault first fault' messages… looking at the source, it almost looks like it's some sort of timing message?
[5:46] <bchrisman> I'm wondering if it indicates some sort of a problem..
[6:41] * ijuz (~ijuz@p4FFF6AB0.dip.t-dialin.net) Quit (Ping timeout: 480 seconds)
[6:50] * ijuz (~ijuz@p4FFF558B.dip.t-dialin.net) has joined #ceph
[7:04] <sage> bchrisman: depends.. if you're just restarting things it may be because the peer addresses haven't all refreshed.
[7:04] <sage> btw, yehuda pushed a change to the libcrypto++ detection. it may or may not solve your issue, not sure
[7:05] <bchrisman> ahh thx
[7:07] <bchrisman> for fo the 'fault first fault'.. it's been going for a few hours since I first initiated it.
[7:08] <sage> which daemon's log?
[7:08] <bchrisman> probably something with my config, I haven't yet gotten a filesystem up and running, so it's not like something was working and now isn't.
[7:08] <sage> oh likely the monitors aren't running, or the ceph.conf addresses don't match the monmap or something
[7:08] <bchrisman> osd log
[7:09] <bchrisman> cmon seems to be running.. so will look at the monmap stuff
[7:17] * f4m8_ is now known as f4m8
[7:26] <bchrisman> Looking through setup instructions, cauthtool -l /data/keyring.mds.test11 shows me key and capabilities… cauthtool —print-key /data/keyring.mds.test11 gives me: 'client.admin not found'… do I need to —gen-key for 'client.admin'?
[7:27] <sage> the clinet.admin key is generated during mkfs.. you should have specified a keyring path during mkcephfs
[7:27] <sage> usually /etc/ceph/keyring.bin
[7:27] <sage> the mds's keyring only needs the mds.test11 key
[7:27] <bchrisman> mkfs generates that key, or is it generated elsewhere?
[7:27] <sage> mkcephfs generates it
[7:28] <bchrisman> ahh… and using cauthtool on that gives the key I need to mount it.
[7:28] <sage> hmm, it hsould be required but isn't. i'll fix thatup.
[7:30] <bchrisman> wiki's a little odd in that regard, as the 'cluster configuration' tells how to mount the filesystem, while the next link down is how to create a new filesystem… I'll probably straighten that up after I'm confident I know how it's all supposed to go.
[7:38] <sage> pushed mkcephfs update on that -k thing (now required)
[7:46] <bchrisman> my mds log gives me these two messages
[7:46] <bchrisman> -- >> pipe(0x7f0ea4002950 sd=5 pgs=0 cs=0 l=0).fault first fault
[7:46] <bchrisman> mds-1.0 ms_handle_connect on
[7:46] <bchrisman> trying to figure out why my mount command returns "can't read superblock", which is (I think) and EIO.
[7:47] <bchrisman> my mount command: mount -t ceph -o name=admin,secret=AQBN1f1MsN91DhAAkQyTfspOTDx95F3sQ2RDXw== /mnt/ceph
[7:47] <bchrisman> with the secret coming from cauthtool --print-key /etc/ceph/admin_keyring
[7:48] <bchrisman> (*an* EIO rather)
[7:48] <bchrisman> will try using the csyn client
[8:05] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) has joined #ceph
[8:54] * alexxy (~alexxy@ Quit (Quit: No Ping reply in 180 seconds.)
[8:54] * alexxy (~alexxy@ has joined #ceph
[8:57] * alexxy (~alexxy@ Quit ()
[8:58] * alexxy (~alexxy@ has joined #ceph
[9:13] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) Quit (Quit: Yoric)
[10:37] * allsystemsarego (~allsystem@ has joined #ceph
[10:54] * Yoric (~David@ has joined #ceph
[13:05] * _FeliXdk (~felix@ has joined #ceph
[13:06] * f4m8_ (~f4m8@lug-owl.de) has joined #ceph
[13:07] * f4m8 (~f4m8@lug-owl.de) Quit (reticulum.oftc.net galapagos.oftc.net)
[13:07] * FeliXdk (~felix@ Quit (reticulum.oftc.net galapagos.oftc.net)
[14:21] * verwilst (~verwilst@router.begen1.office.netnoc.eu) has joined #ceph
[14:21] * verwilst (~verwilst@router.begen1.office.netnoc.eu) Quit (Read error: Connection reset by peer)
[14:21] * verwilst (~verwilst@router.begen1.office.netnoc.eu) has joined #ceph
[16:11] * allsystemsarego_ (~allsystem@ has joined #ceph
[16:11] * allsystemsarego (~allsystem@ Quit (Ping timeout: 480 seconds)
[16:15] * alexxy (~alexxy@ Quit (Remote host closed the connection)
[16:19] <sagewk> bchrismas: 'ceph -s to
[16:19] <sagewk> bchrismas: 'ceph -s' and make sure the mds is up:active
[16:19] * alexxy (~alexxy@ has joined #ceph
[16:23] <sagewk> er, bchrisman: :)
[16:59] * Meths_ (rift@ has joined #ceph
[17:00] <bchrisman> sagewk: [root@test10 ceph]# ceph -s
[17:00] <bchrisman> 2010-12-07 07:53:19.625918 7fbb75c16720 auth: can't open key file(s) ~/.ceph/keyring.bin, /etc/ceph/keyring.bin, .ceph_keyring
[17:00] <bchrisman> 2010-12-07 07:53:22.626826 7fbb725a3710 -- >> pipe(0x7fbb68000b10 sd=4 pgs=0 cs=0 l=0).fault first fault
[17:00] <sagewk> you are probably still missing /etc/ceph/keyring.bin from the mkcephfs
[17:01] <bchrisman> ahh… need to restart those then?
[17:01] <sagewk> mkcephfs needs '-k /etc/ceph/keyring.bin'
[17:01] <bchrisman> think I used a different file.
[17:01] <bchrisman> wasn't sure whether the crypto was required.
[17:01] <bchrisman> ok
[17:01] <sagewk> in that case you can pass -k /the/othe/rpath to ceph -s..
[17:02] <sagewk> it can be turned off (the default i think) but mkcephfs still generates the keys so you can turn it on later
[17:03] * Meths (rift@ Quit (Read error: Operation timed out)
[17:23] * Meths_ is now known as Meths
[17:42] * bchrisman (~Adium@c-24-130-226-22.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[17:44] * lidongyang (~lidongyan@ Quit (Remote host closed the connection)
[17:44] * lidongyang (~lidongyan@ has joined #ceph
[17:51] * greglap (~Adium@ has joined #ceph
[17:57] * lidongyang (~lidongyan@ Quit (Remote host closed the connection)
[17:58] * lidongyang (~lidongyan@ has joined #ceph
[18:11] * lidongyang (~lidongyan@ Quit (Remote host closed the connection)
[18:11] * lidongyang (~lidongyan@ has joined #ceph
[18:12] * verwilst (~verwilst@router.begen1.office.netnoc.eu) Quit (Quit: Ex-Chat)
[18:17] * Meths_ (rift@ has joined #ceph
[18:18] * joshd (~joshd@ip-66-33-206-8.dreamhost.com) has joined #ceph
[18:23] * Meths (rift@ Quit (Ping timeout: 480 seconds)
[18:38] * Yoric (~David@ Quit (Quit: Yoric)
[18:40] * greglap (~Adium@ Quit (Quit: Leaving.)
[18:40] * Meths_ is now known as Meths
[18:43] * bchrisman (~Adium@70-35-37-146.static.wiline.com) has joined #ceph
[18:46] * sjust (~sam@ip-66-33-206-8.dreamhost.com) has joined #ceph
[18:53] * cmccabe1 (~cmccabe@adsl-75-37-25-40.dsl.pltn13.sbcglobal.net) has joined #ceph
[19:23] * cmccabe (~cmccabe@dsl081-243-128.sfo1.dsl.speakeasy.net) Quit (Ping timeout: 480 seconds)
[19:29] <bchrisman> the init script init-ceph seems to be looking for './ceph.conf' instead of /etc/ceph/ceph.conf. I'm guessing the $conf variable should be getting set somewhere… is that distribution-specific? (I'm installing from source as there aren't rhel6 packages yet —or at least.. that I've found)
[19:32] <cmccabe1> bchrisman: how are you invoking it?
[19:33] <bchrisman> well.. for now I've been using ./init-ceph -c /etc/ceph/ceph.conf -a start/stop (which seems to work as far as doing what it's told)
[19:34] <cmccabe1> I think init-ceph behaves differently when invoked as ./init-ceph
[19:34] <bchrisman> … generally I assume in distro-packaging that's copies to /etc/init.d… so it's pretty much expecting that?
[19:34] <cmccabe1> there is that line that starts "if [ `dirname $0` = "." ] && [ $PWD != "/etc/init.d" ];"
[19:35] <bchrisman> ahh I see
[19:35] <bchrisman> I'll move that around then.
[19:35] <cmccabe1> it's to make it easier to use in development
[19:35] <bchrisman> gotchya
[19:36] <bchrisman> heh:
[19:36] <bchrisman> # i hate autoconf:
[19:36] <bchrisman> if [ "/usr/local/etc" = "/usr/etc" ]; then
[19:36] <bchrisman> ETCDIR=/etc/ceph
[19:38] <cmccabe1> yeah, I'd like of like to see that go away and have people just specify --sysconfdir=/etc
[19:39] <cmccabe1> I suppose now that I added --sysconfdir=/etc to the debian packaging scripts it could go away now
[19:40] <cmccabe1> the problem is that people usually go with --prefix=/usr and then autoconf decides sysconfdir=/usr/etc is a good idea (does anyone use that really?)
[19:42] <sagewk> cmccabe1: yeah let's fix that by doing sysconfdir=/etc on our ./configure line in debian/rules and ceph.spec.in
[19:43] <cmccabe1> sagewk: I think I added --sysconfdir=/etc to there when I was doing the "automatically create /etc/ceph dir" change... let me check
[19:46] <cmccabe1> sagewk: oh, wasn't in ceph.spec.in yet.
[19:47] <sagewk> we can remove that crap from those scripts then yeah?
[19:48] <sagewk> that'll make me happy :)
[19:49] <cmccabe1> yeah :)
[19:50] <cmccabe1> let me check if there's any other sysconfdir fudges out there...
[19:50] <sagewk> init-ceph.in mkcephfs.in and cclass.in i think?
[19:58] <cmccabe1> yeah
[19:58] <cmccabe1> well, also cdebugpack
[20:03] <sagewk> skype!
[20:04] * DLange (~DLange@dlange.user.oftc.net) Quit (Remote host closed the connection)
[20:04] * DLange (~DLange@dlange.user.oftc.net) has joined #ceph
[20:09] * ghaskins_mobile (~ghaskins_@66-189-113-47.dhcp.oxfr.ma.charter.com) has joined #ceph
[20:11] * ghaskins_mobile (~ghaskins_@66-189-113-47.dhcp.oxfr.ma.charter.com) Quit ()
[20:27] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) has joined #ceph
[20:35] * gregorg_taf (~Greg@ has joined #ceph
[20:49] <bchrisman> Has anybody built packages for rhel6beta yet?
[20:49] <bchrisman> I couldn't find any… building from spec file right now.
[21:09] <sagewk> not that i know of, let us know how it goes
[21:09] <sagewk> esp with the cryptopp-devel part.. :)
[21:41] * gregorg_taf (~Greg@ Quit (Quit: Quitte)
[22:10] <wido> sagewk: Tried with debug journal = 20, but the OSD's won't crash again...
[22:11] <sagewk> :(
[22:11] <sagewk> i wonder if it's a timing thing
[22:11] <sagewk> or did the workload change?
[22:11] <wido> There was no workload before
[22:12] <wido> They just crashed, second time it was the same
[22:12] <sagewk> maybe set debug journal back to 0 and see if they start crashing again?
[22:14] <wido> I'll try,
[22:16] <wido> well, my OSD's are eating up all my CPU right now, so I expect a crash soon
[22:19] <wido> sagewk: I'll see if I can reproduce, but they won't seem to crash now
[22:20] <wido> btw something else, ever thought about using shmlog for Ceph?
[22:21] <wido> you could have a sepearete process which simply reads the logs the the shm, you could then always write as much debug info as you want, no restart needed on the actual daemon to turn on debugging
[22:21] <sagewk> intersting
[22:21] <cmccabe1> wido: part of the reason why different debugging levels have different impacts is that formatting data to send to the stream is time-consuming
[22:22] <sagewk> a lot of cpu time is burned doing the logging itself, so we'd still want a way to turn it on/off
[22:22] <wido> Isn't the most CPU spend by writing to the disk?
[22:22] <sagewk> which you can do btw with something like ceph mds tell 0 injectargs '--debug-mds 10'
[22:22] <cmccabe1> wido: in some cases we send an extraordinarily large amount of data to the stream if an extraordinarily high debug level is set... like dumping the whole missing set or something
[22:23] <cmccabe1> wido: if you want to test that theory, you can log to /dev/shm
[22:23] <wido> sagewk: I should really write that down somewhere!
[22:23] <wido> cmccabe1: I'll give it a try some day, I think you are spending most time waiting on the disk
[22:26] <cmccabe1> wido: benchmarking is really complex. It's almost certainly not spending very much CPU time at all writing a small amount of logs to disk. But since we're using blocking I/O to write to the disk, that can cause the thread to be blocked waiting for the log file to be written.
[22:26] <cmccabe1> wido: we could solve this problem by not flushing the log file after every log message (i.e. NOT sending std::cout or using std::flush)
[22:27] <cmccabe1> wido: however some people might be uncomfortable with that because it means that, if you crash, you may not see all the log messages that happened.
[22:27] <cmccabe1> er, I meant to say, NOT sending std::endl (which implies flush)
[22:28] <gregaf> Even without that, in order to keep the logs from being a mess, sending error output needs to take a lock which is shared between all threads
[22:28] <gregaf> which slows stuff down if they're ever trying to log at the same time, which would end up happening if all logging was always on
[22:29] <cmccabe1> gregaf: I think we could use thread-local storage to make that lock mostly unecessary
[22:29] <cmccabe1> gregaf: it would only really be needed at the point when we were flushing to the file descriptor
[22:29] <gregaf> mmm, maybe
[22:30] <cmccabe1> gregaf: but it's a good consideration to keep in mind
[22:30] <gregaf> it's an immensely more complicated logging setup though, for very little gain
[22:30] <gregaf> running in production you still aren't going to want high levels of debugging on
[22:30] <gregaf> and at low levels I doubt it matters
[22:31] <cmccabe1> gregaf: the code I have now could easily be extended to use TLS
[22:32] <cmccabe1> gregaf: as for performance... as the old saying goes... benchmark first, optimize later...
[22:32] <cmccabe1> gregaf: I have a sneaky suspicion that that mutex is hurting performance a lot... but until I get a chance to benchmark I won't comment
[22:33] <cmccabe1> oh btw, I implemented something you guys might like.
[22:34] <cmccabe1> Now, when you fire off a high-priority message on dout, it also goes to stderr
[22:34] <wido> tnx, it was just a thought that shm might be an idea. Locks are expensive, indeed, but flushing to disk even more
[22:34] <cmccabe1> (unless stderr has been disabled by daemonize)
[22:35] <cmccabe1> so we can get rid of all the code that manually outputs the same thing to stderr and dout, in scenarios where we haven't daemonized yet.
[22:35] <cmccabe1> and probably un-confuse a few users who missed high-priority messages that someone forgot to manually put on stderr
[22:36] <sagewk> that'll be much cleaner
[22:36] <cmccabe1> thx
[22:37] <wido> i'm going afk, the internet in this hotel (Paris) is really crappy
[22:37] <wido> my terminal lags a lot
[22:37] <sagewk> hmm it does conflate "always log this" with "error, panic"
[22:37] <wido> ttyl
[22:37] <sagewk> wido: ok thanks. let me know if you see that bug again!
[22:39] <cmccabe1> sagewk: yeah, there's only one priority number now
[22:39] <cmccabe1> sagewk: but it does seem like we want the osd.1.log to be a superset of what was on stderr
[22:39] <sagewk> yeah
[22:40] <cmccabe1> sagewk: syslog works kind of similarly... you can set a priority such that any syslog at that priority will appear on your serial console
[22:41] <cmccabe1> turning that on and trying to use the serial console can be an adventure
[22:42] <cmccabe1> mostly because unlike stderr, syslog messages don't necessarily come from a process you started from the terminal
[22:42] <cmccabe1> oh... there's some weird btrfs thing I'm seeing on flab
[22:42] <cmccabe1> Dec 7 13:19:12 sepia1 kernel: [ 1493.578527] =========================================================
[22:42] <cmccabe1> Dec 7 13:19:12 sepia1 kernel: [ 1493.578933] [ INFO: possible irq lock inversion dependency detected ]
[22:42] <cmccabe1> Dec 7 13:19:12 sepia1 kernel: [ 1493.578933] 2.6.37-rc1+ #45
[22:42] <cmccabe1> Dec 7 13:19:12 sepia1 kernel: [ 1493.578933] ---------------------------------------------------------
[22:42] <cmccabe1> Dec 7 13:19:12 sepia1 kernel: [ 1493.578933] cosd/3503 just changed the state of lock:
[22:42] <cmccabe1> Dec 7 13:19:12 sepia1 kernel: [ 1493.578933] (&fs_info->trans_mutex){+.+.-.}, at: [<ffffffffa0020302>] start_transaction+0x5c/0x1b5 [btrfs]
[22:43] <sagewk> those nodes send syslog to flab
[22:43] <sagewk> you can ignore it for now
[22:43] <cmccabe1> k
[23:32] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) Quit (Quit: Yoric)
[23:39] * allsystemsarego_ (~allsystem@ Quit (Quit: Leaving)

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.