#ceph IRC Log

Index

IRC Log for 2012-12-28

Timestamps are in GMT/BST.

[0:00] * l0nk (~alex@87-231-111-125.rev.numericable.fr) has joined #ceph
[0:00] * l0nk (~alex@87-231-111-125.rev.numericable.fr) Quit ()
[0:00] <joshd1> paravoid: that's right, I'd forgotten about that. I'm not sure if it's working currently for the osd, but you can try using it's heap profiler
[0:01] <yasu`> Does Ceph do striping (RAID-0 equiv) ?
[0:02] <yasu`> Placement Groups are for the unit of an object (a file), no ?
[0:02] * Cube (~Cube@c-38-80-203-117.rw.zetabroadband.com) Quit (Quit: Leaving.)
[0:02] <joshd1> paravoid: http://ceph.com/deprecated/Memory_Profiling
[0:02] <paravoid> that /deprecated/ there doesn't inspire me with confidence
[0:03] <joshd1> hehe, yeah. I think that page is still correct though
[0:04] <paravoid> tell heap stats
[0:04] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[0:04] <paravoid> coool!
[0:05] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[0:05] * tonyha (~thariman@202.158.82.17) has joined #ceph
[0:05] * loicd (~loic@magenta.dachary.org) has joined #ceph
[0:08] <paravoid> that's awesome
[0:08] <paravoid> you should definitely copy this over to the new docs
[0:08] <paravoid> I'll file a bug for that
[0:09] <joshd1> thanks
[0:10] <paravoid> oh darn
[0:10] <paravoid> this might be kernel memory
[0:11] * dspano (~dspano@rrcs-24-103-221-202.nys.biz.rr.com) Quit (Quit: Leaving)
[0:15] * ircolle (~ircolle@pool-173-79-213-144.washdc.fios.verizon.net) Quit (Quit: Leaving.)
[0:15] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[0:15] * tonyha (~thariman@202.158.82.17) has joined #ceph
[0:17] * vata (~vata@208.88.110.46) Quit (Quit: Leaving.)
[0:17] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[0:18] * tonyha (~thariman@202.158.82.17) has joined #ceph
[0:20] <joshd1> what kernel/fs?
[0:20] * yasu` (~yasu`@99.23.160.146) Quit (Remote host closed the connection)
[0:25] <paravoid> ubuntu 12.04 LTS, xfs
[0:25] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[0:26] * tonyha (~thariman@202.158.82.17) has joined #ceph
[0:30] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) has joined #ceph
[0:41] <paravoid> 8900736K xfs_inode
[0:41] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[0:41] <paravoid> yay
[0:42] * tonyha (~thariman@202.158.82.17) has joined #ceph
[0:47] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[0:47] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[0:48] * tonyha (~thariman@202.158.82.17) has joined #ceph
[0:50] <joshd1> paravoid: does /proc/$pid/fd/ have tons of entries
[0:50] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[0:50] <paravoid> no, I checked for that already
[0:51] * tonyha (~thariman@202.158.82.17) has joined #ceph
[0:52] <paravoid> xfs_inode seems to be consuming 9GB of RAM
[0:52] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[0:52] <paravoid> xfs_ili another 2GB
[0:52] <paravoid> not very ceph-related I guess
[0:53] * tonyha (~thariman@202.158.82.17) has joined #ceph
[0:53] * natmaka (~nat@gra94-16-83-154-131-43.fbx.proxad.net) has joined #ceph
[0:56] * drokita (~drokita@24-107-180-86.dhcp.stls.mo.charter.com) has joined #ceph
[0:56] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[0:57] <natmaka> Hi! Suggestion: on http://ceph.com/docs/master/install/build-prerequisites/ , in the "On Debian/Squeeze, execute aptitude install ..." subsection, add "pkg-config libcurl4-gnutls-dev" to the provided "aptitude install..." line Those packages are listed above, but missing in the copy/paste'able line
[0:58] * tonyha (~thariman@202.158.82.17) has joined #ceph
[1:04] <natmaka> Under Debian Wheezy (amd64, here) one also need to install "uuid-dev" package (it provides the needed "uuid_parse")
[1:04] * CloudGuy (~CloudGuy@5356416B.cm-6-7b.dynamic.ziggo.nl) Quit (Remote host closed the connection)
[1:07] <glowell> has anyone looked at the kernel gitbuilder ? or rather does anyone mind if I try to fix it myself ?
[1:07] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[1:07] <elder> Go for it glowell
[1:08] <elder> It was getting stuck the other day on 3.8-rc1
[1:08] * tonyha (~thariman@202.158.82.17) has joined #ceph
[1:08] <elder> Some sort of infinite loop on the config file.
[1:10] <joshd1> natmaka: thanks, could you file a bug about that on http://tracker.newdream.net?
[1:15] <natmaka> OK, will do (also: libkeyutils-dev , libaio)
[1:15] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[1:16] * mgalkiewicz (~mgalkiewi@toya.hederanetworks.net) has joined #ceph
[1:16] * tonyha (~thariman@202.158.82.17) has joined #ceph
[1:23] <Kioob> autoreconf: `configure.ac' or `configure.in' is required
[1:23] <Kioob> make: *** [configure-stamp] Error 1
[1:23] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[1:23] <paravoid> joshd1: fwiw, xfs people say that ~20G xfs inode cache is normal if there's otherwise no memory pressure (box has 64G)
[1:23] <Kioob> I'm not able to recompile ceph 0.55.1 for squeeze :(
[1:24] * tonyha (~thariman@202.158.82.17) has joined #ceph
[1:25] <sagewk> joshd: care to look at wip-features, wip-cephx?
[1:25] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[1:25] <glowell> Kioob: Are you building from a tarball or from git source ?
[1:25] <joshd1> paravoid: good to know. nothing stuck out from the heap profiler?
[1:25] <joshd1> sagewk: sure
[1:26] <Kioob> glowell: from git
[1:26] <sagewk> elder: did you touch that sha1 on the gitbuidler yesterday?
[1:26] <paravoid> no, I realized most of the consumption was in kernel memory
[1:26] <paravoid> and started looking at slabinfo etc.
[1:26] <sagewk> elder: oh.. if you did fail/$sha1, ti's proably trying to bisect, which is no better. there is an ignore/ directory.
[1:26] * tonyha (~thariman@202.158.82.17) has joined #ceph
[1:26] <sagewk> or something like that
[1:27] <sagewk> glowell: ^
[1:27] <glowell> That message from autoreconf is suggests the submodules are missing. git submodule update —init should fix that.
[1:27] <glowell> Hi sagewk
[1:28] <Kioob> ok thanks !
[1:28] <sagewk> gitbuilder fixed
[1:29] <glowell> ok
[1:30] <elder> sagewk, yes I did touch it.
[1:31] <elder> I didn't know I could use ignore/
[1:31] <sagewk> i forgot it mattered
[1:31] * noob2 (~noob2@ext.cscinfo.com) Quit (Ping timeout: 480 seconds)
[1:31] <elder> I didn't realize it would automatically bisect. That could be useful!
[1:32] * loicd (~loic@magenta.dachary.org) has joined #ceph
[1:32] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[1:32] <Kioob> glowell: it works, thanks a lot.
[1:32] <glowell> Great.
[1:33] * tonyha (~thariman@202.158.82.17) has joined #ceph
[1:33] <mgalkiewicz> sagewk: Hi, I have experienced problems with upgrading osds from 0.52 to 0.55 https://gist.github.com/raw/4393494/f3ae88406350b74ac6d608b8b75960f85435e85e/gistfile1.txt, dou you think it is the same issue as http://tracker.newdream.net/issues/3467 ?
[1:34] <sagewk> probably not.. that one was on 0.48.2. can you try the latest next branch?
[1:34] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[1:34] <sagewk> is that crash reproducible?
[1:35] <mgalkiewicz> yeah but it crashes some of my clients
[1:35] <mgalkiewicz> on production...
[1:35] * tonyha (~thariman@202.158.82.17) has joined #ceph
[1:36] <sagewk> elder: and yeah, i was testing the feature bit mismatch from that bug. pushing something cleaning it up on the server side too.
[1:36] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[1:37] <elder> OK great.
[1:37] <sagewk> next crashes you clients you mean? or 0.55.1?
[1:37] <mgalkiewicz> 0.55.1
[1:37] <mgalkiewicz> two osds are 0.52 and after starting the one with 0.55.1 it crashes and it is log from this machine
[1:37] <sagewk> there were several issues with 0.55.1 that have been fixed in next, so please try that. it'll get released in a day or two.
[1:38] * tonyha (~thariman@202.158.82.17) has joined #ceph
[1:38] <mgalkiewicz> when do you expect to release debian package?
[1:38] <sagewk> oh, it's a mix of 0.52 an 0.55.1? i wonder if htat is part of hte problem..
[1:38] <paravoid> bobtail is getting released in a day or two?
[1:38] <paravoid> cool :)
[1:38] <sagewk> you can get an autobuilt package now, from http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-basic/ref/next/
[1:41] <mgalkiewicz> sagewk: so do you suggest to run latest build instead of 0.55.1? I still have to keep the rest 2 osds (0.52) before I will be sure that recovery went fine
[1:41] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[1:41] <mgalkiewicz> and then upgrade them as well
[1:41] <sagewk> yeah. so you have 3 osds, 2 0.52, and one 0.55.1 and crashing?
[1:41] <mgalkiewicz> yep
[1:41] <sagewk> the 0.52 ones are stable?
[1:41] <mgalkiewicz> yes
[1:41] <sagewk> then yeah, upgrade the one to latest next.
[1:42] <mgalkiewicz> ok
[1:42] * tonyha (~thariman@202.158.82.17) has joined #ceph
[1:42] <sagewk> if that goes okay, you should be in good shape. if it hits it again, let us know!
[1:42] <Kioob> I have a question about that commit : https://github.com/ceph/ceph/commit/b706945ae9a985761d2aa355307a14b23ea0a3f7 it use a direct syscall to avoid depends from a recent version of libc. But at least on Debian that syscall.h file comes from the libc6 package, not kernel headers
[1:42] <Kioob> is it "normal" ?
[1:43] <sagewk> there is a later commit that also checks for _NR_syncfs, which comes from the kernel headers
[1:43] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[1:43] <Kioob> great !
[1:43] <mgalkiewicz> sagewk: ok I will try
[1:44] <sagewk> thanks!
[1:44] <Kioob> sagewk: ok, I try that
[1:45] * tonyha (~thariman@202.158.82.17) has joined #ceph
[1:46] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) Quit (Quit: Leaving.)
[1:46] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[1:46] * roald (~Roald@87.209.150.214) Quit (Quit: Leaving)
[1:47] * tonyha (~thariman@202.158.82.17) has joined #ceph
[1:48] * scuttlemonkey (~scuttlemo@96-42-136-136.dhcp.trcy.mi.charter.com) Quit (Quit: This computer has gone to sleep)
[1:48] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[1:49] <elder> sagewk, I had a little trouble finding the "failed assertion" because I just removed that in a commit I've posted for review (and that's included in the current state of my tree)
[1:49] <sagewk> elder: which failed assertion?
[1:50] * tonyha (~thariman@202.158.82.17) has joined #ceph
[1:50] <elder> The one you mentioned in the CEPH_CON_CLOSED patch.
[1:50] <sagewk> oh i see
[1:50] <elder> ceph_fault() no longer asserts anything about the socket state.
[1:50] <sagewk> oh, that reminds me. do you have an example of where con->state changes unpredictably?
[1:51] <elder> That was one.
[1:51] <elder> But for that particular thing.
[1:51] <sagewk> iirc it should only change under con->mutex. so we should validate it after taking con->mutex, but everywhere else it will behave in an orderly fashion
[1:51] <sagewk> oooh, right.
[1:51] <elder> Since the underlying socket can close at any time
[1:51] <sagewk> but when that happens, we set the socket state, not the con state
[1:52] <elder> Hang on, let me look again.
[1:52] <sagewk> and catch that at the top of con_work()
[1:56] <elder> You're right, we only set con->state under the mutex. At the top of con_work(), con_sock_closed() is checked.
[1:56] * Etherael (~eric@node-217.pool-125-24.dynamic.totbb.net) Quit (Read error: Connection reset by peer)
[1:57] <elder> What are the possible values of con->state at that point?
[1:57] <paravoid> I'm seeing a bit of a disappointing performance out of storing journals in intel 710 ssds
[1:57] <paravoid> does anyone have any experience with that?
[1:57] <sagewk> probably any state at that point
[1:57] <paravoid> it could be completely unrelated to ceph, but I'm wondering if anyone else has seen that
[1:58] <elder> That's why I removed the assertion about con->state inside ceph_fault().
[1:58] <elder> Because I'm not sure we can say anything about the state. Maybe we can... But I predict at least the majority of states are possible even if not all.
[1:58] <sagewk> hmm.
[1:59] * Etherael (~eric@node-217.pool-125-24.dynamic.totbb.net) has joined #ceph
[1:59] <sagewk> the only other states are CLOSED PREOPEN and STANDBY
[1:59] <sagewk> CLOSED shouldn't happen.
[2:00] <sagewk> PREOPEN shouldn't happen because the socket can't have failed yet
[2:01] <sagewk> i think STANDBY is out too, because we only go there after we process a socket failure. we shouldn't see a second failure bit set on socket state until we try to reopen the socket.
[2:01] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[2:01] <sagewk> did we see this one come up for a case other than Ugis's?
[2:02] <elder> No
[2:02] <elder> I'm tracking back from ceph_fault() now too.
[2:02] * tonyha (~thariman@202.158.82.17) has joined #ceph
[2:03] <elder> We call ceph_fault() if: 1) the socket closed; 2) try_read() returned an error; or 3) try_write() returned an error.
[2:03] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[2:03] <sagewk> my vote is to switch to WARN_ON everywhere, but leave the checks strict until we know that they're too strong
[2:03] <elder> With a WARN_ON() that's a reasonable position...
[2:04] <elder> I thought I'd changed those to WARN_ON() a long time ago. I intended to...
[2:04] <elder> I'm happy to do what you suggest.
[2:04] <elder> I'm going to finish this thought though, tracking back from ceph_fault() because I do prefer tighter constraints if we can define them.
[2:04] <sagewk> k. do you see a case where 1 2 or 3 can happen from another state?
[2:04] <elder> Still looking.
[2:04] <sagewk> cool
[2:04] * tonyha (~thariman@202.158.82.17) has joined #ceph
[2:04] <sagewk> thanks
[2:05] <mgalkiewicz> sagewk: I am afraid 0.55.1-362-gc0fe381-1wheezy crashes the same way
[2:05] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[2:07] * tonyha (~thariman@202.158.82.17) has joined #ceph
[2:07] <sagewk> mgalkiewicz: can you reproduce with 'debug osd = 20' and 'debug ms = 1'?
[2:09] <sagewk> and attach to this bug? http://tracker.newdream.net/issues/3689
[2:09] <sagewk> thanks!
[2:15] <mgalkiewicz> sagewk: ok
[2:15] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[2:15] <elder> sagewk, I agree PREOPEN can't happen, because the state gets set to CONNECTING between that state and calling ceph_tcp_connect()
[2:15] * tonyha (~thariman@202.158.82.17) has joined #ceph
[2:18] * nyeates (~nyeates@pool-173-59-239-231.bltmmd.fios.verizon.net) Quit (Quit: Zzzzzz)
[2:18] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[2:19] * nyeates (~nyeates@pool-173-59-239-231.bltmmd.fios.verizon.net) has joined #ceph
[2:19] * tonyha (~thariman@202.158.82.17) has joined #ceph
[2:23] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) Quit (Quit: Leaving.)
[2:23] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[2:24] * tonyha (~thariman@202.158.82.17) has joined #ceph
[2:27] * scuttlemonkey (~scuttlemo@96-42-136-136.dhcp.trcy.mi.charter.com) has joined #ceph
[2:27] * ChanServ sets mode +o scuttlemonkey
[2:27] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[2:28] <mgalkiewicz> sagewk: log uploaded. Is there any chance that somebody will take a look at this before bobtail release (2 days like you said)?
[2:29] * tonyha (~thariman@202.158.82.17) has joined #ceph
[2:32] * natmaka (~nat@gra94-16-83-154-131-43.fbx.proxad.net) Quit (Quit: Leaving)
[2:32] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[2:32] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) has joined #ceph
[2:33] * tonyha (~thariman@202.158.82.17) has joined #ceph
[2:39] <elder> sagewk, I'm looking at CON_STATE_CLOSED
[2:39] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[2:39] <elder> If ceph_fault() is called on a LOSSYTX channel, what reopens the connection?
[2:39] * tonyha (~thariman@202.158.82.17) has joined #ceph
[2:39] <Kioob> "mount syscall(__NR_syncfs, fd) fully supported", finally, it works :)
[2:40] <Kioob> thanks for your help !
[2:50] <elder> sagewk, I agree that STANDBY is out. It's only set in ceph_fault(), and it *appears* to transition only to CONNECTING, in clear_standby()
[2:51] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) Quit (Quit: Leaving.)
[2:57] * BManojlovic (~steki@243-166-222-85.adsl.verat.net) Quit (Ping timeout: 480 seconds)
[2:57] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[2:57] * tonyha (~thariman@202.158.82.17) has joined #ceph
[2:57] <elder> sagewk, CON_STATE_CLOSED is harder to verify and I've already spent more time than I wanted to on this.
[2:58] <elder> I'm going to put that last assert back in as you suggested.
[3:04] * scuttlemonkey (~scuttlemo@96-42-136-136.dhcp.trcy.mi.charter.com) Quit (Quit: This computer has gone to sleep)
[3:04] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[3:05] * tonyha (~thariman@202.158.82.17) has joined #ceph
[3:09] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[3:09] * Etherael (~eric@node-217.pool-125-24.dynamic.totbb.net) Quit (Read error: Connection reset by peer)
[3:12] * sstan_ (~chatzilla@modemcable130.148-162-184.mc.videotron.ca) Quit (Remote host closed the connection)
[3:15] * Etherael (~eric@node-217.pool-125-24.dynamic.totbb.net) has joined #ceph
[3:15] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[3:15] * tonyha (~thariman@202.158.82.17) has joined #ceph
[3:20] <joao> oh wow
[3:20] <joao> this one is new to me
[3:20] <joao> mon/Monitor.cc: At global scope:
[3:20] <joao> mon/Monitor.cc:2996:2: internal compiler error: Segmentation fault
[3:20] <joao> lol
[3:20] <joao> I think I just crashed g++
[3:20] <elder> Wow.
[3:20] <iggy> you've never worked with Gentoo
[3:21] <iggy> they are like a daily occurence there
[3:21] <joao> I did, but it was a long time ago
[3:21] <joao> fortunately, my memory doesn't hold bad memories from that long ago
[3:22] * iggy shudders... if only he could forget those dark days
[3:22] <joao> second time was the charm ;)
[3:22] <joao> oops
[3:22] <joao> nope
[3:22] <joao> CXXLD ceph-mon
[3:22] <joao> /usr/bin/ld: libmon.a(libmon_a-Monitor.o): bad reloc symbol index (0x2110 >= 0xa6c) for offset 0xc7e8 in section `.debug_ranges'
[3:22] <joao> libmon.a: could not read symbols: Bad value
[3:23] <joao> oh lol
[3:23] <elder> Just copy a few lines in the Makefile, it should work eventually.
[3:25] <joao> first of all, time to push out my work before the desktop thinks it's a good time to crash forever
[3:25] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[3:25] <joao> this kinds of errors makes me all paranoid for some reason
[3:26] <joao> *kind
[3:26] * tonyha (~thariman@202.158.82.17) has joined #ceph
[3:26] <joao> *error
[3:26] <joao> oh boy
[3:28] <elder> *paranoids
[3:28] <elder> :)
[3:34] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) has joined #ceph
[3:37] <joao> sagewk, still around?
[3:37] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[3:38] * tonyha (~thariman@202.158.82.17) has joined #ceph
[3:46] <joao> sagewk, pushed wip-3633 to gh with more informative 'details' on get_health() and stripped out the timeout stuff; also added a patch to support 'json' output on 'health' and 'status', but to be really pretty we'd probably have to refactor the monitor's command handling code (as it is, it just looks like a hack)
[3:46] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[3:46] <joao> and I'm off to bed o/
[3:46] * tonyha (~thariman@202.158.82.17) has joined #ceph
[3:49] * miroslav (~miroslav@173-228-38-131.dsl.dynamic.sonic.net) has joined #ceph
[3:49] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[3:50] * tonyha (~thariman@202.158.82.17) has joined #ceph
[3:50] * miroslav (~miroslav@173-228-38-131.dsl.dynamic.sonic.net) Quit ()
[4:00] * mikedawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) has joined #ceph
[4:00] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[4:00] * tonyha (~thariman@202.158.82.17) has joined #ceph
[4:04] * renzhi (~renzhi@116.226.64.176) has joined #ceph
[4:04] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[4:05] * tonyha (~thariman@202.158.82.17) has joined #ceph
[4:07] <renzhi> hi, I'm trying to add a new osd to the crushmap with the following command: ceph osd crush set 30 osd.30 1.0 pool=default
[4:07] <renzhi> and I get Invalid argument
[4:08] <renzhi> what's wrong in this command?
[4:08] <renzhi> the osd is started already, and it is IN the cluster
[4:17] <mikedawson> renzhi: try ceph osd crush set 30 osd.30 1.0 root=default
[4:18] <mikedawson> see the output of "ceph osd tree" and match the hierarchy
[4:18] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[4:19] <renzhi> I got it, actually need to specify the buckets, I was assuming that ceph would be able to figure out the default from ceph.conf
[4:19] * tonyha (~thariman@202.158.82.17) has joined #ceph
[4:36] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) Quit (Quit: Leaving.)
[4:36] * tonyha (~thariman@202.158.82.17) Quit (Read error: Connection reset by peer)
[4:37] * tonyha (~thariman@202.158.82.17) has joined #ceph
[4:57] * tonyha (~thariman@202.158.82.17) Quit (Quit: tonyha)
[5:01] * joshd1 (~jdurgin@2602:306:c5db:310:9943:7039:6371:bc19) Quit (Quit: Leaving.)
[5:15] * calebamiles1 (~caleb@c-107-3-1-145.hsd1.vt.comcast.net) has joined #ceph
[5:18] * calebamiles (~caleb@c-107-3-1-145.hsd1.vt.comcast.net) Quit (Ping timeout: 480 seconds)
[5:28] * scuttlemonkey (~scuttlemo@96-42-136-136.dhcp.trcy.mi.charter.com) has joined #ceph
[5:28] * ChanServ sets mode +o scuttlemonkey
[5:31] * maxiz (~pfliu@202.108.130.138) has joined #ceph
[5:34] * Cube (~Cube@c-38-80-203-117.rw.zetabroadband.com) has joined #ceph
[5:34] * scuttlemonkey (~scuttlemo@96-42-136-136.dhcp.trcy.mi.charter.com) Quit (Quit: This computer has gone to sleep)
[6:30] * Karcaw_ (~evan@68-186-68-219.dhcp.knwc.wa.charter.com) Quit (Quit: leaving)
[6:47] * Cube (~Cube@c-38-80-203-117.rw.zetabroadband.com) Quit (Quit: Leaving.)
[6:50] * renzhi (~renzhi@116.226.64.176) Quit (Quit: Leaving)
[6:55] * drokita (~drokita@24-107-180-86.dhcp.stls.mo.charter.com) Quit (Quit: Leaving.)
[6:55] * mikedawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) Quit (Ping timeout: 480 seconds)
[6:58] * Cube (~Cube@c-38-80-203-117.rw.zetabroadband.com) has joined #ceph
[7:27] * Karcaw (~evan@68-186-68-219.dhcp.knwc.wa.charter.com) has joined #ceph
[7:28] * Cube (~Cube@c-38-80-203-117.rw.zetabroadband.com) Quit (Quit: Leaving.)
[7:48] * maxiz (~pfliu@202.108.130.138) Quit (Quit: Ex-Chat)
[8:03] * madkiss (~madkiss@178.188.60.118) has joined #ceph
[8:04] * madkiss (~madkiss@178.188.60.118) Quit ()
[8:31] * low (~low@188.165.111.2) has joined #ceph
[8:45] * Cube (~Cube@c-38-80-203-117.rw.zetabroadband.com) has joined #ceph
[8:53] * yasu` (~yasu`@99.23.160.146) has joined #ceph
[8:53] * yasu` (~yasu`@99.23.160.146) Quit (Remote host closed the connection)
[8:53] * Etherael (~eric@node-217.pool-125-24.dynamic.totbb.net) Quit (Read error: Connection reset by peer)
[8:59] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) has joined #ceph
[9:37] * Etherael (~eric@node-217.pool-125-24.dynamic.totbb.net) has joined #ceph
[9:50] <alexxy> hi all
[9:50] <alexxy> is there any best practice about ceph cluster setup?
[9:51] <alexxy> we have here 18 nodes
[9:51] <alexxy> that should work as osd and as clients
[9:51] <alexxy> kernel is 3.7
[9:51] * loicd (~loic@magenta.dachary.org) has joined #ceph
[9:51] <alexxy> ceph is 0.55.1
[9:51] <alexxy> sometimes osd crashed
[9:51] <alexxy> or nodes goes into deadlock
[10:07] * agh (~2ee79308@2600:3c00::2:2424) has joined #ceph
[10:08] <agh> Hello to all
[10:08] <agh> I am killing myself with CephFS !
[10:09] <agh> I need a replacement for NFS in a distributed way. So CephFS is good
[10:09] <agh> I have a test cluster of 6 OSD nodes, with 3 to 10 disks each
[10:09] <agh> Everythins is working fine on ONE client : mount -t ceph ... /mnt/cephfs
[10:10] <agh> perfs are not so bad, it's cool
[10:10] <agh> BUT BUT... When i mount the same pool on a second client... ooops it does not work anymore
[10:11] <agh> The second client can mount, and it works, but then, the first one become stuck. Not possible to do a "ls" or even to unmount the CephFS mount point
[10:11] <agh> Do you have any idea ?
[10:12] * gaveen (~gaveen@112.135.32.18) has joined #ceph
[10:13] <agh> I am killing myself with CephFS ! I need a replacement for NFS in a distributed way. So CephFS is good I have a test cluster of 6 OSD nodes, with 3 to 10 disks each Everythins is working fine on ONE client : mount -t ceph ... /mnt/cephfs perfs are not so bad, it's cool BUT BUT... When i mount the same pool on a second client... ooops it does not work anymore The second client can mount, and it works, but then, the first one become stuck. Not possible to do a
[10:32] * madkiss (~madkiss@178.188.60.118) has joined #ceph
[10:37] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) has left #ceph
[10:58] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[11:21] * loicd (~loic@178.20.50.225) has joined #ceph
[11:22] * Cube (~Cube@c-38-80-203-117.rw.zetabroadband.com) Quit (Quit: Leaving.)
[11:23] * Cube (~Cube@c-38-80-203-117.rw.zetabroadband.com) has joined #ceph
[11:25] * MarcoA (5158e06e@ircip2.mibbit.com) has joined #ceph
[11:25] <MarcoA> Hello
[11:26] <MarcoA> I have some problems adding new osd to my cluster
[11:27] <MarcoA> 1 mon, 1 mds, 3 osd on 1 host
[11:27] <MarcoA> I want to add the 4th osd on the host
[11:28] <MarcoA> the cluster is deployed via mkcephfs
[11:30] <MarcoA> the 4th osd is prepared with "ceph-disk-prepare /dev/sde /dev/sda" (sda is a ssd with all the journals)
[11:30] <MarcoA> then i followed these instuctions: http://ceph.com/docs/master/rados/operations/add-or-rm-osds/
[11:31] <MarcoA> but I can't see the 4th up
[11:33] <MarcoA> ceph -s (http://pastebin.com/Ef8xmQzJ)
[11:34] <MarcoA> ceph osd tree (http://pastebin.com/PuTfCtN1)
[11:35] <MarcoA> ceph osd dump (http://pastebin.com/8v5WqGCG)
[11:40] <MarcoA> ceph.conf (http://pastebin.com/zYH4uTGa)
[11:42] <MarcoA> Crush Map (http://pastebin.com/mjcn4vta)
[11:53] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[11:53] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[11:56] <joao> MarcoA, did you start the osd?
[11:56] * MarcoA (5158e06e@ircip2.mibbit.com) Quit (Quit: http://www.mibbit.com ajax IRC Client)
[11:56] <joao> timing is everything
[12:03] * Kioob`Taff (~plug-oliv@local.plusdinfo.com) Quit (Quit: Leaving.)
[12:14] * morse (~morse@supercomputing.univpm.it) Quit (Remote host closed the connection)
[12:17] * madkiss (~madkiss@178.188.60.118) Quit (Ping timeout: 480 seconds)
[12:17] * morse (~morse@supercomputing.univpm.it) has joined #ceph
[12:19] * gaveen (~gaveen@112.135.32.18) Quit (Remote host closed the connection)
[12:41] * MarcoA (5158e06e@ircip2.mibbit.com) has joined #ceph
[12:47] * Psi-jack (~psi-jack@psi-jack.user.oftc.net) Quit (Read error: Operation timed out)
[12:48] * Psi-jack (~psi-jack@psi-jack.user.oftc.net) has joined #ceph
[12:52] <agh> helpp to all
[12:53] <agh> i have a lot of these errors :
[12:53] <agh> 2012-12-28 11:02:46.601739 osd.8 [WRN] slow request 173.834635 seconds old, received at 2012-12-28 10:59:52.767008: osd_sub_op(client.5856.1:100241 0.31d 5e910b1d/1000000000f.00000f71/head//0 [] v 57'16 snapset=0=[]:[] snapc=0=[]) v7 currently started
[12:53] <agh> any idea ?
[13:09] * madkiss (~madkiss@178.188.60.118) has joined #ceph
[13:09] * madkiss1 (~madkiss@178.188.60.118) has joined #ceph
[13:17] * madkiss (~madkiss@178.188.60.118) Quit (Ping timeout: 480 seconds)
[13:26] * renzhi (~renzhi@116.226.64.176) has joined #ceph
[13:35] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) has joined #ceph
[13:43] <renzhi> hello, I'm getting some nasty lock issue in the librados. Here is the trace:
[13:43] <renzhi> http://pastebin.com/PZd6znA9
[13:44] <renzhi> whenever this happens, the whole application locks up for a few seconds, and then librados gets out of it, and everything continues again. After a few minutes, it would get into the same situation again.
[13:45] <renzhi> Sometimes, it would get into this situations several times in a row. And there's nothing we can do but wait for it to get out of it.
[13:47] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) Quit (Quit: Leaving.)
[13:47] * ScOut3R (~ScOut3R@1F2E59B1.dsl.pool.telekom.hu) has joined #ceph
[13:52] <nhm> renzhi: ugh, it's far too early. :)
[13:52] <renzhi> nhm: I know, but it's late for us, been struggling for a day now
[13:54] <nhm> renzhi: sadly I don't actually know much about invoking librados. I wonder if look at the rados bench src would provide any useful clues if you guys are invoking librados differently.
[13:54] <nhm> renzhi: not saying there isn't a bug, maybe there is a work around.
[13:56] <renzhi> nhm: yes, we are trying to find a work-around solution too, the code where this issue happens looks innocent enough, all it does is getting a lock, write some data, unlock, and be done.
[13:56] <renzhi> this is in librados::IoCtxImpl::write()
[13:57] <renzhi> we just don't know the rados internal well enough to find an easy way out
[13:59] <nhm> renzhi: this is probably going to be a blind-leading-the-blind sort of situation, but I can try to flounder around a bit before someone who knows what they are doing shows up. ;)
[13:59] <renzhi> nhm: any hint would be very much appreciated
[13:59] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) has joined #ceph
[14:00] * Vjarjadian_ (~IceChat77@5ad6d005.bb.sky.com) Quit (Quit: Never put off till tomorrow, what you can do the day after tomorrow)
[14:00] <nhm> renzhi: mind sending me a copy of the code in question? mark.nelson@inktank.com
[14:01] <renzhi> nhm: hang on
[14:08] <renzhi> nhm: just sent the email
[14:09] <nhm> renzhi: yeah, looking it over
[14:09] <renzhi> thanks a lot
[14:09] <nhm> renzhi: no problem, but I'll warn you right now that you've probably looked at librados more than I have at this point. :)
[14:10] <nhm> renzhi: My guess is we'll probably need to have someone who knows the librados internals dig into this to actually gain any real traction.
[14:12] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) Quit (Quit: Leaving.)
[14:13] <renzhi> nhm: not a problem, you probably might get a better grab of the issue.
[14:14] <mgalkiewicz> agh: still there?
[14:14] <renzhi> The real annoying thing is that, when this happens, it puts the application in real stand still, and nothing would work, until it gets out of it
[14:16] <nhm> renzhi: any idea what the state of the OSDs are when it happens?
[14:16] <nhm> renzhi: IE, are they still accepting write ops? Are buffers full, etc?
[14:19] <renzhi> nhm: yes, the OSDs are accepting write ops, but they are doing some remapping, as we just added a few more osds.
[14:19] <renzhi> nhm: but we can read/write
[14:20] <nhm> renzhi: Is this just a recent occurance or has it been happening all 6 months?
[14:20] <renzhi> nhm: just happened this morning (China time), at around 9am
[14:21] <renzhi> so that's like 12 hours ago now.
[14:21] <nhm> renzhi: wasn't happening previously?
[14:22] <renzhi> no
[14:25] <renzhi> nhm: this issue looks like it's having issue connecting to ceph in the backend. We had a nasty crash two weeks ago, and we had never been able to recover the cluster. That's the only time we saw something similar.
[14:30] <MarcoA> I'm back.
[14:30] <MarcoA> joao: Yes, the osd is started
[14:32] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[14:32] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[14:33] <MarcoA> i'm thinking that ceph does not see the 4th osd as a member of the cluster. Am I wrong?
[14:33] <nhm> renzhi: going afk for a while, my kids just woke up and I have to get them breakfast/dressed.
[14:34] * scuttlemonkey (~scuttlemo@96-42-136-136.dhcp.trcy.mi.charter.com) has joined #ceph
[14:34] * ChanServ sets mode +o scuttlemonkey
[14:34] <renzhi> nhm: ok
[14:40] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) has joined #ceph
[14:43] <janos> aww yeah, east coaster
[14:43] <janos> ;)
[14:51] * MarcoA (5158e06e@ircip2.mibbit.com) has left #ceph
[14:51] * MarcoA (~aroldi@81.88.224.110) has joined #ceph
[14:54] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[14:54] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) Quit (Quit: Leaving.)
[14:54] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[14:55] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) has joined #ceph
[15:00] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[15:00] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[15:06] * gaveen (~gaveen@112.135.32.18) has joined #ceph
[15:16] <sstan> Good Morning!
[15:16] <janos> hail
[15:21] * slang (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) Quit (Remote host closed the connection)
[15:22] * slang (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) has joined #ceph
[15:22] * SkyEye (~gaveen@112.135.28.246) has joined #ceph
[15:23] * ScOut3R (~ScOut3R@1F2E59B1.dsl.pool.telekom.hu) Quit (Remote host closed the connection)
[15:24] * XSBen (~XSBen@195.220.156.20) Quit (Ping timeout: 480 seconds)
[15:25] * XSBen (~XSBen@195.220.156.20) has joined #ceph
[15:25] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) Quit (Quit: Leaving.)
[15:29] * gaveen (~gaveen@112.135.32.18) Quit (Ping timeout: 480 seconds)
[15:30] * mikedawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) has joined #ceph
[15:52] * XSBen (~XSBen@195.220.156.20) Quit (Ping timeout: 480 seconds)
[15:52] * XSBen (~XSBen@195.220.156.20) has joined #ceph
[15:53] <agh> mgalkiewicz: yes, sorry
[15:56] * dspano (~dspano@rrcs-24-103-221-202.nys.biz.rr.com) has joined #ceph
[15:57] * SkyEye is now known as gaveen
[16:01] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) has joined #ceph
[16:04] * KindOne (~KindOne@h138.181.130.174.dynamic.ip.windstream.net) has joined #ceph
[16:07] * danieagle (~Daniel@177.97.249.46) has joined #ceph
[16:12] * The_Bishop (~bishop@2001:470:50b6:0:8dd8:8e2f:a8f8:f189) has joined #ceph
[16:15] * ircolle (~ircolle@pool-173-79-213-144.washdc.fios.verizon.net) has joined #ceph
[16:19] * elder (~elder@c-71-195-31-37.hsd1.mn.comcast.net) Quit (Quit: Leaving)
[16:41] * elder (~elder@c-71-195-31-37.hsd1.mn.comcast.net) has joined #ceph
[16:41] * ChanServ sets mode +o elder
[16:46] * drokita (~drokita@199.255.228.10) has joined #ceph
[16:48] * mtk (oL2yrKfTDW@panix2.panix.com) Quit (Remote host closed the connection)
[16:48] * mtk (0gNPzIZze3@panix2.panix.com) has joined #ceph
[16:52] * drokita1 (~drokita@199.255.228.10) has joined #ceph
[16:53] * Karcaw (~evan@68-186-68-219.dhcp.knwc.wa.charter.com) Quit (Quit: leaving)
[16:55] * drokita (~drokita@199.255.228.10) Quit (Ping timeout: 480 seconds)
[16:58] * drokita (~drokita@199.255.228.10) has joined #ceph
[17:00] * drokita2 (~drokita@199.255.228.10) has joined #ceph
[17:00] * drokita1 (~drokita@199.255.228.10) Quit (Ping timeout: 480 seconds)
[17:06] * drokita (~drokita@199.255.228.10) Quit (Ping timeout: 480 seconds)
[17:16] <sstan> will libvirt integrate with RADOS when running the XEN hypervisor?
[17:19] <nhm> renzhi: I ended up forwarding your email to a couple of other guys on our team. We'll see what they have to say about it.
[17:19] <sstan> sstan: apparently not : http://libvirt.org/storage.html#StorageBackendRBD
[17:20] <nhm> sstan: yeah, I think for xen we are still limited to the kernel implementation.
[17:21] <sstan> nhm: we have to rbd map , right? Is there another way?
[17:23] <nhm> sstan: not that I know of, but Josh is the right guy to ask about this.
[17:23] <sstan> ok
[17:24] <nhm> elder or dmick might know too.
[17:24] <sstan> the only other option I can think of has something to do with FUSE
[17:25] <sstan> But that would only be good for cephFS, i.e. nothing to do with RBD ?
[17:25] <nhm> yeah, the fuse client is for cephfs.
[17:26] <sstan> yeah.. I really have to figure out how to make the rbd module work. Trying to make it work on SLES (kernel 3.0)
[17:37] * ircolle (~ircolle@pool-173-79-213-144.washdc.fios.verizon.net) Quit (Quit: Leaving.)
[17:37] * drokita2 (~drokita@199.255.228.10) Quit (Ping timeout: 480 seconds)
[17:39] <elder> sstan, you need to check with joshd when (and if) he gets online today.
[17:39] <elder> 3.0 is a pretty old kernel, though the port may not be hard.
[17:40] <sstan> okay thanks! I'll write a message if I figure something out. But it seems that SLES will be integrating ceph
[17:40] <wido> sstan: It's still on my whishlist to implement kernel RBD at some point in libvirt
[17:40] <sstan> https://www.suse.com/support/update/announcement/2012/suse-su-20121016-1.html
[17:41] <wido> problem is that you need to have libvirt do the mapping when the guest starts and do a unmap when it stops
[17:41] <wido> there is no such mechanism for "hooks" now in libvirt
[17:41] <sstan> ah that's how it works
[17:41] <sstan> are there any lock mechanisms? do they expire ?
[17:41] <wido> you can use libvirt to manage your RBD pool, create images, delete them, etc
[17:41] * ircolle (~ircolle@pool-173-79-213-144.washdc.fios.verizon.net) has joined #ceph
[17:41] <wido> locks inside libvirt?
[17:42] <sstan> something that locks the image so that it cannot be used by two machines on the network
[17:42] <sstan> I guess libvirt should do that
[17:42] <wido> sstan: No, RBD has to handle that
[17:43] * drokita (~drokita@199.255.228.10) has joined #ceph
[17:43] <wido> RBD nowadays supports locking
[17:43] <wido> http://ceph.com/docs/master/man/8/rbd/
[17:43] <sstan> ah there are locks : )
[17:44] <wido> Indeed
[17:45] <sstan> so ... one could use that feature by writing a resource agent (pacemaker, etc.)
[17:52] * low (~low@188.165.111.2) Quit (Quit: bbl)
[17:53] <wido> sstan: Yes, you can use that to lock images
[17:54] <sstan> the only problem remaining is that, apparently, one should NOT run OSDs and RBD mappings on the same machine ...
[17:59] <drokita> Is it normal for osds to report as not-up in 'ceph -s' if they are not in the cluster?
[18:00] <drokita> for example, this is the output of ceph -s: osdmap e340: 4 osds: 2 up, 2 in
[18:00] <drokita> However, I just started the other 2 osds on the partner server
[18:01] <sagewk> drokita: sometimes it takes a bit for them to replay their journal and then join.
[18:01] <drokita> ok
[18:01] <sagewk> or you may have the 'noup' flag set.. ceph osd dump | grep flags
[18:01] <sagewk> or they may have problems authenticating or something :)
[18:02] <drokita> Bingo... they just joined
[18:02] * Gugge-47527 (gugge@kriminel.dk) has joined #ceph
[18:02] <drokita> Patience is a virtue
[18:11] * Cube (~Cube@c-38-80-203-117.rw.zetabroadband.com) Quit (Ping timeout: 480 seconds)
[18:13] * dspano (~dspano@rrcs-24-103-221-202.nys.biz.rr.com) Quit (Read error: Operation timed out)
[18:13] * dspano (~dspano@rrcs-24-103-221-202.nys.biz.rr.com) has joined #ceph
[18:16] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) Quit (Quit: Leaving.)
[18:19] <Gugge-47527> 000
[18:22] * joshd1 (~jdurgin@2602:306:c5db:310:3da4:6b57:1f57:d0ff) has joined #ceph
[18:22] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) has joined #ceph
[18:24] * jbd_ (~jbd_@34322hpv162162.ikoula.com) has left #ceph
[18:32] <MarcoA> hello, can I ask for a help?
[18:32] <MarcoA> I'd like to add a osd to my test cluster
[18:33] <MarcoA> but i can't get the osd up
[18:33] <MarcoA> it stays "in" for 5 minutes, the ceph marks out
[18:34] <MarcoA> the cluster is deployed via mkcephfs
[18:34] <MarcoA> 1 mon, 1 mds, 3 osd on 1 host
[18:34] <MarcoA> the 4th osd is prepared with "ceph-disk-prepare /dev/sde /dev/sda" (sda is a ssd with all the journals)
[18:34] <MarcoA> then i followed these instuctions: http://ceph.com/docs/master/rados/operations/add-or-rm-osds/
[18:34] <MarcoA> but I can't see the 4th up
[18:34] <MarcoA> ceph -s (http://pastebin.com/Ef8xmQzJ)
[18:35] <MarcoA> ceph.conf (http://pastebin.com/zYH4uTGa)
[18:35] <MarcoA> ceph osd tree (http://pastebin.com/PuTfCtN1)
[18:35] <MarcoA> ceph osd dump (http://pastebin.com/8v5WqGCG)
[18:35] <MarcoA> Crush Map (http://pastebin.com/mjcn4vta)
[18:35] * Cube (~Cube@c-38-80-203-117.rw.zetabroadband.com) has joined #ceph
[18:36] * jbarbee (17192e61@ircip3.mibbit.com) has joined #ceph
[18:36] <sstan> MarcoA: maybe you just need to wait until the map propagates
[18:38] * jbarbee_ (~chatzilla@c-98-226-73-253.hsd1.in.comcast.net) has joined #ceph
[18:39] <MarcoA> sstan: how much? I've tried 6 hours ago, and nothing has changed
[18:39] * jbarbee (17192e61@ircip3.mibbit.com) has left #ceph
[18:39] * danieagle (~Daniel@177.97.249.46) Quit (Quit: Inte+ :-) e Muito Obrigado Por Tudo!!! ^^)
[18:39] * jbarbee_ is now known as jbarbee_1
[18:40] * jbarbee_1 is now known as jbarbee1
[18:41] <MarcoA> sstan: i have done the prepare and the firts attempt to join the cluster this morning
[18:42] <MarcoA> sstan: and now, after 6 hours, i'm still blocked
[18:42] <joshd1> MarcoA: is the ceph-osd process with -i 3 still running? if not, is there anything in its log file (/var/log/ceph/ceph-osd.3.log)?
[18:42] <MarcoA> sstan: yes, the process -i 3 is running
[18:44] <joshd1> is there anything in its log file?
[18:46] <MarcoA> this is the log, just fired a "ceph restart osd.3" (http://pastebin.com/8i4qk5yx)
[18:48] <renzhi> nhm: thanks
[18:53] <joshd1> MarcoA: how about your monitor's log? it looks like the monitor may not be allowing the osd to boot
[18:55] <MarcoA> yes, i'm just lookint at the mon log
[18:55] <MarcoA> it writes only this line:
[18:55] <MarcoA> 2012-12-28 18:54:34.890913 7fb691c55700 0 mon.a@0(leader).osd e41 preprocess_boot on fsid 9a5db6d4-ec8d-4bb2-89ce-ed9dc7262f21 != 56ac9684-635b-4840-b361-ea1af5337656
[18:55] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[18:55] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[18:56] <MarcoA> i have the line "fsid = 9a5db6d4-ec8d-4bb2-89ce-ed9dc7262f21" in my conf
[18:57] <joshd1> that would be the problem
[18:58] <MarcoA> do i have to remove that line?
[18:58] <joshd1> where did that fsid come from?
[18:58] <MarcoA> uuidgen -r
[18:59] <joshd1> yeah, remove that line and re-initialize that osd with ceph-disk-prepare
[19:00] <MarcoA> tryin'
[19:02] <MarcoA> ceph-disk-prepare: OSD preparation error: getting cluster uuid from configuration failed
[19:04] <joshd1> add --cluster-uuid 56ac9684-635b-4840-b361-ea1af5337656
[19:04] <joshd1> when you run ceph-disk-prepare
[19:08] * gaveen (~gaveen@112.135.28.246) Quit (Remote host closed the connection)
[19:09] <MarcoA> YES!
[19:09] <MarcoA> the on is up and in
[19:09] <MarcoA> the osd is up and in
[19:10] <MarcoA> Ok guys thank you very much
[19:11] <joshd1> that uuid is an identifier for the entire cluster, so it has to match what the monitors have (visible via 'ceph fsid')
[19:12] <MarcoA> ok, imho this have to be underlined in the docs.
[19:13] <joshd1> agreed, there should be docs for ceph-disk-prepare
[19:13] <MarcoA> yes.
[19:13] <joshd1> it's not necessary with other setup methods
[19:13] <joshd1> it's there as a safety check so you don't accidentally add an osd to the wrong cluster
[19:14] <MarcoA> ok. Thanks again
[19:14] <joshd1> no problem
[19:14] * MarcoA (~aroldi@81.88.224.110) has left #ceph
[19:18] * CloudGuy (~CloudGuy@5356416B.cm-6-7b.dynamic.ziggo.nl) has joined #ceph
[19:21] <jbarbee1> joshd1: i am working with mdawson on a folsom openstack cluster backed by ceph
[19:22] <jbarbee1> we have been trying to create a volume to boot from and take advantage of cow with cinder
[19:23] <jbarbee1> I found that i am starting with 'format 2' image, but after the image downloads and imports into ceph volumes pool, it is in 'format 1'
[19:24] * mtk (0gNPzIZze3@panix2.panix.com) Quit (Remote host closed the connection)
[19:24] * mtk (~mtk@ool-44c35983.dyn.optonline.net) has joined #ceph
[19:25] <jbarbee1> looking at var/log/cinder-volume.log, it appears that the format flag is not being used when the image imports,
[19:25] <jbarbee1> rbd import --pool volumes /tmp/tmpMmgBfP volume-1ffafb7e-dc54-44ce-8afd-349ac233d84b execute /usr/lib/python2.7/dist-packages/cinder/utils.py:156
[19:25] <exec> folks, is there any release date of bobtail?
[19:25] <jbarbee1> Do you think this is a config issue or a bug in the cinder python script?
[19:27] * nyeates (~nyeates@pool-173-59-239-231.bltmmd.fios.verizon.net) Quit (Quit: Zzzzzz)
[19:27] * nwat2 (~Adium@c-50-131-197-174.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[19:27] <jbarbee1> joshd1: here is a paste of the step by step, http://pastebin.com/cn0PNeVn
[19:28] * nwat (~Adium@c-50-131-197-174.hsd1.ca.comcast.net) has joined #ceph
[19:28] * nwat (~Adium@c-50-131-197-174.hsd1.ca.comcast.net) Quit ()
[19:29] <joshd1> jbarbee1: config issue. it only uses 'rbd import' if it thinks it can't do a clone
[19:32] <joshd1> jbarbee1: does glance have show_image_direct_url=True in the default section of its config?
[19:34] <jbarbee1> joshd1: yes
[19:35] <joshd1> jbarbee1: ah, I see the error is in the log: Unable to read image rbd://a95153ac-510f-46c2-b244-62b5413e0999/images/19ee4596-ad12-4e90-8fe3-b41456ca44bd/snap
[19:35] <joshd1> what does 'ceph auth list' show for osd caps for the client cinder is using (client.volumes?)
[19:36] <jbarbee1> client.volumes
[19:36] <jbarbee1> key: AQDgUc9QICmMAhAAmUK2JxPYWRFRALO5QeQPLA==
[19:36] <jbarbee1> caps: [mon] allow r
[19:36] <jbarbee1> caps: [osd] allow rwx pool=volumes, allow rx pool=images
[19:37] <jbarbee1> i do not see a client.cinder , assuming client.volumes is the one
[19:39] * scuttlemonkey_ (~scuttlemo@96-42-136-136.dhcp.trcy.mi.charter.com) has joined #ceph
[19:40] <joshd1> jbarbee1: it's client.volumes if you have CEPH_ARGS="--id volumes" set in cinder-volume's environment
[19:41] <jbarbee1> that is how we have it set
[19:43] * scuttlemonkey (~scuttlemo@96-42-136-136.dhcp.trcy.mi.charter.com) Quit (Read error: Operation timed out)
[19:44] * scuttlemonkey (~scuttlemo@96-42-136-136.dhcp.trcy.mi.charter.com) has joined #ceph
[19:44] * ChanServ sets mode +o scuttlemonkey
[19:44] <joshd1> jbarbee1: this is actually a bug in ceph. the watch operation used to be incorrectly flagged as a read. it is now flagged as a write. However, 'rbd info' needs the write capability, since it does a watch when it opens the rbd image
[19:45] <joshd1> jbarbee1: the work around is to give client.volumes write access to the images pool as well
[19:46] <jbarbee1> joshd1: thx, i will try that
[19:49] * scuttlemonkey_ (~scuttlemo@96-42-136-136.dhcp.trcy.mi.charter.com) Quit (Ping timeout: 480 seconds)
[19:57] <iggy> exec: I think I heard a few days yesterday, but no quotes
[19:57] <sagewk> just caught an important issue affecting mixed-version clusters, need to fix that first.
[19:58] <sagewk> http://tracker.newdream.net/issues/3689
[20:00] * Ryan_Lane (~Adium@c-67-160-217-184.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[20:01] * scuttlemonkey_ (~scuttlemo@96-42-136-136.dhcp.trcy.mi.charter.com) has joined #ceph
[20:01] * calebamiles1 (~caleb@c-107-3-1-145.hsd1.vt.comcast.net) Quit (Remote host closed the connection)
[20:01] * jbarbee1 (~chatzilla@c-98-226-73-253.hsd1.in.comcast.net) Quit (Quit: ChatZilla 0.9.89 [Firefox 17.0.1/20121128204232])
[20:02] <psiekl> sagewk, btw: thanks for supporting multi-version clusters. it was one of the things that really made an impression on us when we saw your talk at LISA; it's such a refreshing change from the usual "shut everything down to upgrade" stance that's so ops-unfriendly.
[20:02] <sagewk> psiekl: :)
[20:07] * scuttlemonkey (~scuttlemo@96-42-136-136.dhcp.trcy.mi.charter.com) Quit (Ping timeout: 480 seconds)
[20:15] * scuttlemonkey (~scuttlemo@96-42-136-136.dhcp.trcy.mi.charter.com) has joined #ceph
[20:15] * ChanServ sets mode +o scuttlemonkey
[20:17] <janos> ughm i think i'm gonna wait until the bobtail rpm/repo update
[20:17] <janos> i just made my own 0.55.1 rom's and local repo
[20:17] <janos> (fedora17)
[20:18] <janos> i can't get a stable cluster going
[20:18] <janos> rom's/rpm's
[20:18] * vata (~vata@208.88.110.46) has joined #ceph
[20:21] * scuttlemonkey_ (~scuttlemo@96-42-136-136.dhcp.trcy.mi.charter.com) Quit (Ping timeout: 480 seconds)
[20:21] <dspano> psiekl: I second that.
[20:22] <denken> one of twelve storage nodes was rebooted in our cluster today... within 60 seconds of that event, 10 other OSD's in the cluster terminated with "./common/Mutex.h: 89: FAILED assert(nlock == 0)"
[20:22] <denken> its the second time ive seen it happen
[20:22] <denken> argonaut
[20:23] <denken> 10 others, spread across six other storage chassis, that is
[20:25] * scuttlemonkey_ (~scuttlemo@96-42-136-136.dhcp.trcy.mi.charter.com) has joined #ceph
[20:29] * CloudGuy (~CloudGuy@5356416B.cm-6-7b.dynamic.ziggo.nl) Quit (Remote host closed the connection)
[20:30] * scuttlemonkey (~scuttlemo@96-42-136-136.dhcp.trcy.mi.charter.com) Quit (Ping timeout: 480 seconds)
[20:30] * CloudGuy (~CloudGuy@5356416B.cm-6-7b.dynamic.ziggo.nl) has joined #ceph
[20:32] <joshd1> denken: if you have logs from that (even just the full back trace) it'd be useful
[20:33] <denken> yeah ive got logs, core dumps, etc.
[20:33] <denken> i was about to open an issue unless you'd like otherwise
[20:34] <joshd1> that'd be great
[20:34] <denken> the second strange bit is that once those terminated OSD's were restarted, the cluster lost quorum for 9min 42secs
[20:36] * astalsi (~astalsi@0001aef0.user.oftc.net) Quit (Remote host closed the connection)
[20:37] <joshd1> anything strange happening to the nodes the monitors are on?
[20:38] <joshd1> like high load that would stop them from responding quickly?
[20:38] <denken> one mon was down, being moved to a dedicated node. im wondering if all the IO generated from bringing those 10 osds up/in + the osd's that were on the rebooted node caused it
[20:38] <denken> yeah thats what im thinking
[20:39] <denken> one of three mon's i should say. one of the remaining two mons was on a dedicated chassis, and the other was living alongside 14 osd's.
[20:40] <joshd1> could be high load on the one next to 14 osds if you restarted 10 other osds at once
[20:43] <sagewk> mgalkiewicz: wip-3689 has a fix for that mixed cluster issue.. please test if you can!
[20:44] * noob2 (~noob2@ext.cscinfo.com) has joined #ceph
[20:44] <noob2> you guys are generally good with running monitors on the same hosts as osd's right?
[20:46] <joshd1> noob2: yeah, usually it's not an issue
[20:46] <noob2> cool :)
[20:46] * madkiss1 (~madkiss@178.188.60.118) Quit (Quit: Leaving.)
[20:47] <noob2> i'll break up the monitors into different rows just to be safe
[20:47] * Ryan_Lane (~Adium@216.38.130.165) has joined #ceph
[20:47] <noob2> joshd1: any idea if bobtail is going to contain support to expand page groups?
[20:48] <noob2> placement groups i mean*
[20:48] * dmick (~dmick@2607:f298:a:607:1530:43d8:4550:1cb4) has joined #ceph
[20:48] <sagewk> it does, but the feature is slightly obscured until we have invested more time in testing
[20:48] * Etherael1 (~eric@node-37t.pool-125-24.dynamic.totbb.net) has joined #ceph
[20:49] <noob2> ok thanks. i'm expecting this cluster i am building to double or triple in size in a few years so having that ability is great
[20:51] * BManojlovic (~steki@243-166-222-85.adsl.verat.net) has joined #ceph
[20:54] * agh (~2ee79308@2600:3c00::2:2424) Quit (Quit: TheGrebs.com CGI:IRC (Session timeout))
[20:54] * Etherael (~eric@node-217.pool-125-24.dynamic.totbb.net) Quit (Ping timeout: 480 seconds)
[20:54] * madkiss (~madkiss@178.188.60.118) has joined #ceph
[20:55] * madkiss (~madkiss@178.188.60.118) Quit ()
[20:56] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) Quit (Quit: Leaving.)
[20:57] * madkiss (~madkiss@178.188.60.118) has joined #ceph
[21:01] * madkiss (~madkiss@178.188.60.118) Quit ()
[21:02] * scuttlemonkey (~scuttlemo@96-42-136-136.dhcp.trcy.mi.charter.com) has joined #ceph
[21:02] * ChanServ sets mode +o scuttlemonkey
[21:02] <denken> joshd1: 3692
[21:02] <denken> if you want the core dumps or anything else, just say the word
[21:02] * nyeates (~nyeates@pool-173-59-239-231.bltmmd.fios.verizon.net) has joined #ceph
[21:03] * Vjarjadian (~IceChat77@5ad6d005.bb.sky.com) has joined #ceph
[21:03] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) has joined #ceph
[21:04] * darkfader (~floh@188.40.175.2) Quit (Quit: leaving)
[21:08] * scuttlemonkey_ (~scuttlemo@96-42-136-136.dhcp.trcy.mi.charter.com) Quit (Ping timeout: 480 seconds)
[21:09] * CloudGuy (~CloudGuy@5356416B.cm-6-7b.dynamic.ziggo.nl) Quit (Remote host closed the connection)
[21:09] * loicd (~loic@178.20.50.225) Quit (Ping timeout: 480 seconds)
[21:09] * calebamiles (~caleb@c-107-3-1-145.hsd1.vt.comcast.net) has joined #ceph
[21:09] * CloudGuy (~CloudGuy@5356416B.cm-6-7b.dynamic.ziggo.nl) has joined #ceph
[21:10] * CloudGuy (~CloudGuy@5356416B.cm-6-7b.dynamic.ziggo.nl) Quit (Remote host closed the connection)
[21:11] * scuttlemonkey (~scuttlemo@96-42-136-136.dhcp.trcy.mi.charter.com) Quit (Read error: Operation timed out)
[21:14] * darkfader (~floh@188.40.175.2) has joined #ceph
[21:22] * mgalkiewicz (~mgalkiewi@toya.hederanetworks.net) Quit (Ping timeout: 480 seconds)
[21:23] * The_Bishop (~bishop@2001:470:50b6:0:8dd8:8e2f:a8f8:f189) Quit (Ping timeout: 480 seconds)
[21:26] * tziOm (~bjornar@ti0099a340-dhcp0628.bb.online.no) has joined #ceph
[21:31] * The_Bishop (~bishop@2001:470:50b6:0:5d57:9151:a3b5:129d) has joined #ceph
[21:34] * mgalkiewicz (~mgalkiewi@toya.hederanetworks.net) has joined #ceph
[21:41] <dmick> hey Kioob, you around?
[21:47] <dmick> Kioob: http://tracker.newdream.net/issues/3693
[21:50] * ChanServ sets mode +o dmick
[21:53] * miroslav (~miroslav@173-228-38-131.dsl.dynamic.sonic.net) has joined #ceph
[21:54] * miroslav (~miroslav@173-228-38-131.dsl.dynamic.sonic.net) Quit ()
[21:56] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) Quit (Quit: Leaving.)
[21:57] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) has joined #ceph
[21:57] * loicd (~loic@magenta.dachary.org) has joined #ceph
[22:07] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) has joined #ceph
[22:13] <jefferai> joshd1: any thoughts on enabling those parameters on the kernel for virtio? Should I file a feature-request bug?
[22:18] * miroslav (~miroslav@173-228-38-131.dsl.dynamic.sonic.net) has joined #ceph
[22:18] <paravoid> what's the structure of the objects in .rgw?
[22:18] <paravoid> I got a hexdump and figured out the pool name and the numeric prefix for objects under .rgw.buckets
[22:18] <paravoid> but it'd be nice to be able to properly parse it
[22:20] * The_Bishop (~bishop@2001:470:50b6:0:5d57:9151:a3b5:129d) Quit (Quit: Wer zum Teufel ist dieser Peer? Wenn ich den erwische dann werde ich ihm mal die Verbindung resetten!)
[22:25] * CloudGuy (~CloudGuy@5356416B.cm-6-7b.dynamic.ziggo.nl) has joined #ceph
[22:34] * ircolle (~ircolle@pool-173-79-213-144.washdc.fios.verizon.net) Quit (Quit: Leaving.)
[22:42] <sagewk> paravoid: some of them are in cls/rgw/cls_rgw_types.h
[22:42] <sagewk> others are in rgw/*.h
[22:43] <paravoid> hm
[22:43] <paravoid> I was wondering about the one that has an object per bucket, named with the name of the bucket
[22:43] <paravoid> I think .rgw only has those
[22:45] <dmick> http://article.gmane.org/gmane.comp.file-systems.ceph.devel/8301/match=rgw+object+names has some info too
[22:45] <paravoid> thanks
[22:45] <dmick> http://thread.gmane.org/gmane.comp.file-systems.ceph.devel/8300 better link
[22:46] <paravoid> on an unrelated not very ceph related note, any ideas on why ceph would easily starve intel 710 ssds with only 160 iops/15mb per second?
[22:47] <paravoid> these ssds are spec'ed and benchmarked for orders of magnitudes better performance than that
[22:47] <paravoid> I'm wondering if the ceph workload is special in some way
[22:47] <paravoid> journal workload that is
[22:53] <Kioob> thanks dmick :) Yes, I finally fixed my conf, but the error message doesn't help
[22:54] <dmick> Kioob: yep. hopefully we can improve the parser
[22:56] * NightDog (~karl@ti0131a340-dhcp0997.bb.online.no) has joined #ceph
[22:58] * miroslav (~miroslav@173-228-38-131.dsl.dynamic.sonic.net) Quit (Quit: Leaving.)
[23:09] * noob2 (~noob2@ext.cscinfo.com) Quit (Quit: Leaving.)
[23:09] * houkouonchi-home (~linux@fios.houkouonchi.jp) has joined #ceph
[23:20] * madkiss (~madkiss@178.188.60.118) has joined #ceph
[23:22] * madkiss (~madkiss@178.188.60.118) Quit ()
[23:23] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[23:23] * loicd (~loic@magenta.dachary.org) has joined #ceph
[23:27] * madkiss (~madkiss@178.188.60.118) has joined #ceph
[23:28] * NightDog (~karl@ti0131a340-dhcp0997.bb.online.no) Quit (Quit: This computer has gone to sleep)
[23:33] * The_Bishop (~bishop@e179005132.adsl.alicedsl.de) has joined #ceph
[23:34] * tziOm (~bjornar@ti0099a340-dhcp0628.bb.online.no) Quit (Remote host closed the connection)
[23:45] * madkiss (~madkiss@178.188.60.118) Quit (Quit: Leaving.)
[23:50] * mikedawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.