#ceph IRC Log


IRC Log for 2012-10-11

Timestamps are in GMT/BST.

[0:00] <nwl> i note that the mount IP address for Ceph FS has to be the same as what the mon IP address reports back - which means you can't mount a cluster which is behind a NAT box. I wonder how that impacts using Load Balancers..
[0:02] <joshd> nwl: generally pseudo-random placement is a replacement for load balancers in ceph. the monitors aren't in the data path
[0:05] <nwl> joshd: specifically referring to this: [ 353.904526] libceph: wrong peer, want, got
[0:06] * dspano (~dspano@rrcs-24-103-221-202.nys.biz.rr.com) Quit (Quit: Leaving)
[0:07] <andreask> I'd use a ipsec/vpn connection anywas to access my data via public networks ..
[0:07] * dty (~derek@testproxy.umiacs.umd.edu) Quit (Ping timeout: 480 seconds)
[0:07] * slang (~slang@ace.ops.newdream.net) Quit (Quit: slang)
[0:08] * PerlStalker (~PerlStalk@ Quit (Quit: rcirc on GNU Emacs 24.2.1)
[0:10] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[0:10] * loicd (~loic@magenta.dachary.org) has joined #ceph
[0:16] * Cube1 (~Adium@cpe-76-95-223-199.socal.res.rr.com) Quit (Quit: Leaving.)
[0:22] * synapsr (~synapsr@c-98-234-186-68.hsd1.ca.comcast.net) has joined #ceph
[0:25] <nwl> trying to add a new OSD and it's failing with this:
[0:25] <nwl> [pid 1415] open("/etc/ceph/keyring,/etc/ceph/keyring.bin", O_WRONLY|O_CREAT|O_TRUNC, 0600) = -1 ENOENT (No such file or directory)
[0:25] <nwl> can open() take two file names like that? never seen that before in a strace
[0:26] <gregaf> yeah, that's incorrect
[0:26] <gregaf> are you getting that out of the monitor, or something else?
[0:26] <mikeryan> probably an error in ceph.conf
[0:26] <joao> that was fixed yesterday
[0:26] <joao> it's on the monitor
[0:27] <joao> we were simply passing 'g_conf->keyring' to KeyRing::load(), instead of single file
[0:27] <gregaf> I'm just not sure why the monitor would be opening the keyring to add an OSD, is why I'm asking
[0:27] <nwl> i was trying to add a new OSD - seems like a standard behaviour so how come people haven't been hit by this before?
[0:27] <nwl> gregaf: it's not the monitor. i am running this on a brand new OSD host
[0:27] <joao> oh
[0:28] <joao> well, looks like this must exist on the monitor too then; same bug for sure
[0:28] <gregaf> could easily be the same sort of thing
[0:28] <joao> yes
[0:28] <nwl> i ran: ceph-osd -i 2 --mkfs --mkkey --osd-data=/var/lib/ceph/osd/ceph-2 --debug_osd 10
[0:28] <nwl> (and shouldn't that be --debug-osd :-)
[0:29] <joao> nwl, as a workaround, I believe you just have to specify the correct keyring file on ceph.conf or pass it --keyring=/path/to/keyring
[0:30] <nwl> joao: thanks
[0:30] <joao> nwl, I *think* that either '_' or '-' is okay given that the function that handles them will always convert (iirc) '-' to '_'
[0:30] <nwl> just looks odd on the display to see a mix of - and _
[0:31] <joao> yeah, usually it should be '--debug-osd'
[0:31] * gregaf1 (~Adium@2607:f298:a:607:2482:18da:8103:1bd8) has joined #ceph
[0:31] <joao> at least because it's prettier
[0:31] <joao> :)
[0:32] <nwl> joao: once i generate the key, i need to copy it over to a monitor and register it there, is that correct?
[0:33] * mtk (~mtk@ool-44c35bb4.dyn.optonline.net) Quit (Remote host closed the connection)
[0:33] * tryggvil (~tryggvil@163-60-19-178.xdsl.simafelagid.is) Quit (Ping timeout: 480 seconds)
[0:34] * miroslavk (~miroslavk@173-228-38-131.dsl.dynamic.sonic.net) has joined #ceph
[0:34] * synapsr_ (~synapsr@c-98-234-186-68.hsd1.ca.comcast.net) has joined #ceph
[0:34] * synapsr (~synapsr@c-98-234-186-68.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[0:35] * sagelap (~sage@ has joined #ceph
[0:37] * gregaf2 (~Adium@ has joined #ceph
[0:38] <nwl> so it looks like --osd-data has become a mandatory field for ceph-osd . is this a receng thing as the docs don't mention it and it breaking start-up scripts..
[0:38] <nwl> s/receng/recent/
[0:39] * sagelap1 (~sage@2600:1010:b02f:a5ea:f061:2419:36b3:eb4e) has joined #ceph
[0:39] <joshd> it has a default, but you've always needed to create the directory yourself
[0:40] <joshd> /var/lib/ceph/osd/$cluster-$id by default
[0:42] * mtk (~mtk@ool-44c35bb4.dyn.optonline.net) has joined #ceph
[0:42] * miroslavk (~miroslavk@173-228-38-131.dsl.dynamic.sonic.net) Quit (Ping timeout: 480 seconds)
[0:43] * tryggvil_ (~tryggvil@163-60-19-178.xdsl.simafelagid.is) has joined #ceph
[0:43] <nwl> joshd: the directory exists already and it is populated. when i run /etc/init.d/ceph -a start , ceph-osd is being invoked again and i get an error about not specifying the -osd-data flag
[0:44] * gregaf1 (~Adium@2607:f298:a:607:2482:18da:8103:1bd8) Quit (Ping timeout: 480 seconds)
[0:44] * sagelap (~sage@ Quit (Ping timeout: 480 seconds)
[0:49] * Leseb_ (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) has joined #ceph
[0:49] * Leseb (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) Quit (Read error: Connection reset by peer)
[0:49] * Leseb_ is now known as Leseb
[0:50] * sagelap1 (~sage@2600:1010:b02f:a5ea:f061:2419:36b3:eb4e) Quit (Ping timeout: 480 seconds)
[0:50] * gregaf2 (~Adium@ Quit (Quit: Leaving.)
[0:59] * miroslavk (~miroslavk@173-228-38-131.dsl.dynamic.sonic.net) has joined #ceph
[0:59] <joshd> nwl: does your ceph.conf specify 'osd data = ' anywhere?
[1:00] <nwl> joshd: no
[1:01] * gregaf1 (~Adium@ has joined #ceph
[1:01] <joshd> nwl: what's the exact error message?
[1:03] <nwl> joshd: it's not an error message so much as something not being invoked properly
[1:03] <nwl> joshd: from a monitor, i am doing this:
[1:03] <nwl> service ceph -a start
[1:03] <nwl> when it gets to my new OSD:
[1:03] <nwl> === osd.2 ===
[1:03] <nwl> Starting Ceph osd.2 on domU-12-31-39-0A-30-56.compute-1.internal...
[1:03] <nwl> 2012-10-10 23:03:05.492683 7f83a49be780 must specify '--osd-data=foo' data path
[1:03] <nwl> 2012-10-10 23:03:05.492922 7f83a49be780 usage: ceph-osd -i osdid [--osd-data=path] [--osd-journal=path] [--mkfs] [--mkjournal] [--convert-filestore]
[1:05] * Cube1 (~Adium@ has joined #ceph
[1:07] <joshd> there must be a code path you're hitting that's not using the defaults for some reason
[1:07] <joshd> could you find out exactly what command's being run there?
[1:08] <nwl> joshd: i can't seem to tell. strace isn't showing anything and there is nothing in the logs
[1:09] <nwl> joshd: who maintains the service scripts?
[1:09] * nwatkins (~nwatkins@soenat3.cse.ucsc.edu) Quit (Remote host closed the connection)
[1:10] * rweeks (~rweeks@c-98-234-186-68.hsd1.ca.comcast.net) Quit (Quit: rweeks)
[1:10] <joshd> nwl: if they're init scripts, add set -x to the beginning to trace them
[1:11] <joshd> nwl: Tv's been working on the upstart scripts
[1:15] <nwl> joshd: i can't tell what the service script is calling to determine the ssh commands it is sending
[1:19] <joshd> oh, right, that'll be in ceph_common.sh
[1:19] <joshd> I don't remember where that's installed, but 'dpkg -L ceph' should tell you
[1:20] <nwl> it is in /usr/lib/ceph
[1:20] <nwl> i am surprised that i am hitting these issues - they don't seem like edge cases
[1:21] <joshd> me too. I'm guessing there's something strange about your setup
[1:21] <joshd> which docs are you following?
[1:21] * gregaf1 (~Adium@ Quit (Quit: Leaving.)
[1:22] <nwl> joshd: the ones on ceph.com
[1:22] <joshd> these ones? http://ceph.com/docs/master/config-cluster/
[1:23] <nwl> joshd: http://ceph.com/docs/master/cluster-ops/add-or-rm-osds/#adding-an-osd-manual
[1:24] <nwl> joshd: my config looks pretty vanilla. I have two OSDs and was trying to add a third. Not exactly rocket science
[1:29] <joshd> nwl: are you using the ubuntu quantal packages, or the precise ones? the osd data error could be because older ceph had a different default, and the docs were written for the new default
[1:30] <joshd> nwl: you should have ceph --version reporting 0.48.2
[1:31] * synapsr_ (~synapsr@c-98-234-186-68.hsd1.ca.comcast.net) Quit (Remote host closed the connection)
[1:34] <nwl> it's all on precise so 0.41
[1:35] <gregaf> oh, that would be why
[1:35] <gregaf> you should upgrade to argonaut; install instructions are in the docs you've already found
[1:37] * synapsr (~synapsr@c-98-234-186-68.hsd1.ca.comcast.net) has joined #ceph
[1:38] <nwl> hold up so argonaut was not in precise?
[1:38] <joshd> no, it's in quantal, and will probably be available via the cloud archive to precise
[1:39] <nwl> doh
[1:39] <nwl> well, what a huge massive fail of an assumption !
[1:39] <nwl> however, it has been really useful to try and get someting broken to work - taught me a lot
[1:40] * gregorg (~Greg@ Quit (Read error: Connection reset by peer)
[1:40] * gregorg (~Greg@ has joined #ceph
[1:41] <nwl> aha so hear is the problem: i did have the argonaut apt line but it installed only ceph not ceph-common
[1:41] * Leseb (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) Quit (Quit: Leseb)
[1:42] * gregorg_taf (~Greg@ has joined #ceph
[1:42] * gregorg (~Greg@ Quit (Read error: Connection reset by peer)
[1:44] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) Quit (Ping timeout: 480 seconds)
[1:47] <dmick> nwl: that's odd; surely ceph depends on ceph-common?..
[1:48] <nwl> dmick: that's what i thought
[1:50] <joshd> it doesn't depend on the exact version of ceph-common though
[1:51] <nwl> joshd: so that's what got me. i installed the precise packages, then added the ceph repo, then did apt-get install ceph
[1:51] <nwl> joshd: and it didn't grab the new ceph-common too. that's a problem
[1:53] * dty (~derek@pool-71-178-175-208.washdc.fios.verizon.net) has joined #ceph
[1:53] * synapsr (~synapsr@c-98-234-186-68.hsd1.ca.comcast.net) Quit (Remote host closed the connection)
[1:56] * joey_ (~joey@97-118-114-147.hlrn.qwest.net) Quit (Quit: Lost terminal)
[1:59] <joshd> nwl: i don't think that's one that should be fixed by packages, but by documenting that you should make sure you're using >=0.48 client and server side for those instructions
[2:01] <nwl> joshd: i agree that we should have that warning but ceph and ceph-common should handle their dependency versions well too.
[2:01] * Leseb (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) has joined #ceph
[2:02] * Leseb (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) Quit ()
[2:07] * maelfius (~mdrnstm@ Quit (Quit: Leaving.)
[2:07] * gregorg_taf (~Greg@ Quit (Read error: Connection reset by peer)
[2:08] * gregorg (~Greg@ has joined #ceph
[2:08] * gregorg (~Greg@ Quit (Read error: Connection reset by peer)
[2:09] * brambles (xymox@grip.espace-win.org) Quit (Ping timeout: 480 seconds)
[2:10] * gregorg (~Greg@ has joined #ceph
[2:11] * dty (~derek@pool-71-178-175-208.washdc.fios.verizon.net) Quit (Read error: Connection reset by peer)
[2:12] * dty (~derek@pool-71-178-175-208.washdc.fios.verizon.net) has joined #ceph
[2:12] * Ludo (~Ludo@falbala.zoxx.net) Quit (Ping timeout: 480 seconds)
[2:29] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[2:32] * loicd (~loic@magenta.dachary.org) has joined #ceph
[2:32] * jlogan2 (~Thunderbi@ Quit (Quit: jlogan2)
[2:32] * jlogan (~Thunderbi@2600:c00:3010:1:559b:87b:4b8:bab6) has joined #ceph
[2:38] * davidz1 (~Adium@2607:f298:a:607:bca7:9cd4:ac20:12ab) has joined #ceph
[2:42] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[2:44] * loicd (~loic@magenta.dachary.org) has joined #ceph
[2:54] * Ryan_Lane (~Adium@ Quit (Quit: Leaving.)
[3:02] * jlogan (~Thunderbi@2600:c00:3010:1:559b:87b:4b8:bab6) Quit (Quit: jlogan)
[3:03] * jlogan (~Thunderbi@2600:c00:3010:1:559b:87b:4b8:bab6) has joined #ceph
[3:06] * slang (~slang@ace.ops.newdream.net) has joined #ceph
[3:07] * Cube1 (~Adium@ Quit (Quit: Leaving.)
[3:20] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[3:20] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[3:28] * Ryan_Lane (~Adium@120.sub-70-197-3.myvzw.com) has joined #ceph
[3:36] * Ryan_Lane (~Adium@120.sub-70-197-3.myvzw.com) Quit (Quit: Leaving.)
[3:37] * miroslavk (~miroslavk@173-228-38-131.dsl.dynamic.sonic.net) Quit (Quit: Leaving.)
[3:49] * aliguori (~anthony@cpe-70-123-130-163.austin.res.rr.com) Quit (Quit: Ex-Chat)
[3:50] * aliguori (~anthony@cpe-70-123-130-163.austin.res.rr.com) has joined #ceph
[3:53] * chutzpah (~chutz@ Quit (Quit: Leaving)
[4:00] * jlogan (~Thunderbi@2600:c00:3010:1:559b:87b:4b8:bab6) Quit (Ping timeout: 480 seconds)
[4:03] * jlogan (~Thunderbi@2600:c00:3010:1:559b:87b:4b8:bab6) has joined #ceph
[4:05] * davidz1 (~Adium@2607:f298:a:607:bca7:9cd4:ac20:12ab) Quit (Quit: Leaving.)
[4:09] * yoshi (~yoshi@p37219-ipngn1701marunouchi.tokyo.ocn.ne.jp) has joined #ceph
[4:19] * jlogan (~Thunderbi@2600:c00:3010:1:559b:87b:4b8:bab6) Quit (Ping timeout: 480 seconds)
[4:41] * miroslavk (~miroslavk@c-98-248-210-170.hsd1.ca.comcast.net) has joined #ceph
[5:02] * dty (~derek@pool-71-178-175-208.washdc.fios.verizon.net) Quit (Quit: dty)
[5:02] * dty (~derek@pool-71-178-175-208.washdc.fios.verizon.net) has joined #ceph
[5:15] * joshd (~joshd@2607:f298:a:607:221:70ff:fe33:3fe3) Quit (Quit: Leaving.)
[5:29] * cattelan (~cattelan@2001:4978:267:0:21c:c0ff:febf:814b) Quit (Ping timeout: 480 seconds)
[5:39] * cattelan (~cattelan@2001:4978:267:0:21c:c0ff:febf:814b) has joined #ceph
[5:43] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[5:43] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[5:47] * renzhi (~renzhi@ has joined #ceph
[5:48] * cattelan (~cattelan@2001:4978:267:0:21c:c0ff:febf:814b) Quit (Ping timeout: 480 seconds)
[5:57] * cattelan (~cattelan@2001:4978:267:0:21c:c0ff:febf:814b) has joined #ceph
[5:58] * jlogan (~Thunderbi@ has joined #ceph
[6:04] * dmick (~dmick@2607:f298:a:607:b9ec:1767:e06a:3f9a) Quit (Quit: Leaving.)
[6:14] * miroslavk (~miroslavk@c-98-248-210-170.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[6:15] * grant (~grant@202-173-147-27.mach.com.au) has joined #ceph
[6:16] <grant> Hey guys, I'd like to include the following btrfs options when using mkcephfs - can you please help me with where to specify?
[6:16] <grant> mkfs.btfs options: -l 16k -n 16k
[6:17] <mikeryan> grant: i'm not sure, but i can take a look
[6:21] <mikeryan> it looks like we don't support extra btrfs options
[6:21] <mikeryan> i'm not positive, but that's my reading of the code
[6:22] <mikeryan> most of the devs are out, it's 9 PM US west coast time
[6:23] <mikeryan> you can try emailing ceph-devel@vger.kernel.org, and i'll be sure to ask tomorrow during business hours
[6:26] <grant> I saw the commands above in the official blog as well :)
[6:27] <grant> I'm wondering if I am best to simply mkcephfs and then mkfs.btrfs with options afterward?
[6:27] <grant> Thank you though, mikeryan :)
[6:28] <mikeryan> grant: which blog post?
[6:28] <grant> http://ceph.com/community/ceph-performance-part-1-disk-controller-write-throughput/
[6:29] <grant> Mark specifically mentions using those options down in the Test Setup section just before the results
[6:32] <mikeryan> ah i see, that was in a test environment
[6:33] <mikeryan> i believe he was using custom scripts to bring up ceph for the sake of reliable measurements
[6:36] <grant> But then reliable won't be reproduceable and the benchmark results not based on real-world performance?
[6:36] <grant> Anyway, I'll ask on the mailing list and see how we go! :)
[6:37] <mikeryan> yep, sorry i couldn't help!
[6:37] <grant> No worries, thanks for trying!
[6:38] * dty (~derek@pool-71-178-175-208.washdc.fios.verizon.net) Quit (Ping timeout: 480 seconds)
[6:40] * jlogan (~Thunderbi@ Quit (Read error: Connection reset by peer)
[6:40] * jlogan (~Thunderbi@ has joined #ceph
[6:42] * miroslavk (~miroslavk@173-228-38-131.dsl.dynamic.sonic.net) has joined #ceph
[7:10] * grifferz (~andy@specialbrew.392abl.bitfolk.com) has joined #ceph
[7:20] * tryggvil_ (~tryggvil@163-60-19-178.xdsl.simafelagid.is) Quit (Quit: tryggvil_)
[7:29] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[7:38] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[7:38] * Ryan_Lane (~Adium@c-67-160-217-184.hsd1.ca.comcast.net) has joined #ceph
[7:50] * kermin (ca037809@ircip2.mibbit.com) has joined #ceph
[7:51] <kermin> how to build ceph from source code ? can anyone help me out
[7:52] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[7:52] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[7:52] <mikeryan> kermin: there's a build guide in the tarball
[7:52] <mikeryan> is there a reason you don't want to use your distro's package?
[7:53] <kermin> sorry to say but I am very new to this stuff. dont knw wats distro package :(
[7:53] <kermin> i was following http://ceph.com/docs/master/source/building-ceph/ link to build it
[7:54] <kermin> so setup git , then built ceph documentation , got the tarball as well
[7:54] <kermin> but yum install rpm-build rpmdevtools is not executing properly
[7:55] <mikeryan> ah, we provide RPM packages for ceph
[7:56] <kermin> ok
[7:57] <mikeryan> actually scratch that, i don't think we publish them yet
[7:57] <mikeryan> but we will soon
[7:57] <mikeryan> you can wait about 10 hours for the rest of the devs to wake up, or i can try to help you build from source
[7:58] <kermin> i only have problem executing "yum install rpm-build rpmdevtools"
[7:58] <kermin> rest all required steps I dod
[7:58] <kermin> *did
[7:58] <mikeryan> what error do you get?
[7:58] <kermin> No package rpm-build available. No package rpmdevtools available. Nothing to do
[8:00] <mikeryan> kermin: what distro are you using?
[8:00] <mikeryan> ubuntu? red hat?
[8:00] <kermin> ubuntu
[8:00] <mikeryan> ah, we definitely provide packages for ubuntu
[8:00] <mikeryan> what version?
[8:00] <kermin> Ubuntu 12.04.1
[8:00] * yoshi (~yoshi@p37219-ipngn1701marunouchi.tokyo.ocn.ne.jp) Quit (Remote host closed the connection)
[8:00] <mikeryan> http://ceph.com/docs/master/install/debian/
[8:00] <mikeryan> follow those directions, let me know if you run into any trouble
[8:01] <mikeryan> you only need to follow the part under "add stable release packages"
[8:01] <kermin> ok.. yeah
[8:02] <kermin> i executed all 3 cmds ..
[8:03] <mikeryan> ah, now you can install it with this command
[8:03] <mikeryan> sudo apt-get update && sudo apt-get install ceph
[8:04] <kermin> by doing so I will get ceph installed directly .. its not installed from tarball that I have downloaded ?
[8:04] <mikeryan> that's right
[8:04] <kermin> I have 3 node cluster that i have developed using these steps . right now I was trying to build from source code
[8:05] <mikeryan> right, we generally recommend using the distro packages unless you have a good reason to use source
[8:06] <kermin> Our plan is to use specific source so that all experiments are carried out with same version .
[8:07] <mikeryan> ok, well let's look at source again
[8:07] <mikeryan> http://ceph.com/docs/master/source/building-ceph/
[8:07] <mikeryan> you gave me that link, but i don't see rpmbuild in there
[8:07] <kermin> http://ceph.com/docs/master/source/build-packages/
[8:08] <kermin> here in this page "yum install rpm-build rpmdevtools" is not working
[8:08] <mikeryan> ah yes, you don't need to follow those instructions
[8:08] <mikeryan> those are for building an RPM for redhat systems
[8:08] <mikeryan> since you're using ubuntu, that's not necessary
[8:09] <kermin> But I thought that watever tarball I have downloaded will be used in cmd "rpmbuild -tb ~/rpmbuild/SOURCES/ceph-<version>.tar.gz"
[8:10] <mikeryan> you won't run the rpmbuild command, since that builds an RPM
[8:10] <mikeryan> you run this command to build a .deb:
[8:10] <mikeryan> sudo dpkg-buildpackage
[8:11] <kermin> tail: cannot open `debian/changelog' for reading: No such file or directory dpkg-buildpackage: error: tail of debian/changelog gave error exit status 1
[8:11] <mikeryan> you have to untar your source tarball and run that command from inside the source directory
[8:12] <kermin> ok sir
[8:16] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[8:16] <kermin> I untar it in /home/atish/ then executer the cmd you told inside "/home/atish/ceph-0.48.2argonaut/src" but still same error is shown
[8:17] <mikeryan> run it in the ceph-0.48.2argonaut dir
[8:17] <kermin> no change , same error
[8:19] <kermin> shall i follow the procedure told in readme file inside tarball ?
[8:21] <mikeryan> well that's a problem with our tarball
[8:21] <mikeryan> what url did you get the tarball from?
[8:22] <kermin> http://ceph.com/download/
[8:22] <kermin> what file should I follow .tar.bz2 or only .tar.gz is fine ?
[8:23] <mikeryan> ah, we don't ship the debian build scripts with the normal tarball
[8:24] <mikeryan> yes, try following the README and INSTALL files from the tarball
[8:24] <mikeryan> you won't be able to build a .deb, but you'll still build from source
[8:24] <mikeryan> tar.bz2 and tar.gz have the same contents, so whatever you downloaded is fine
[8:25] <kermin> I just want to use consistent version to experiment , That is my goal .
[8:25] <kermin> And even it is good learning for me to build it from source code
[8:25] * lxo (~aoliva@lxo.user.oftc.net) Quit (Quit: later)
[8:26] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[8:26] <kermin> anyways where are you from , sir ?
[8:28] <mikeryan> i work for inktank in california
[8:28] <kermin> as all other developers will be up after 10 hrs . i was just curious
[8:28] <mikeryan> most of the developers are in our time zone
[8:28] <mikeryan> UTC-0800
[8:28] <mikeryan> sorry, UTC-0700 right now
[8:29] <kermin> hmm yeah most developers are from your time zone .Anyways thanks for your help .
[8:29] <kermin> u also need to take rest I believe
[8:29] <mikeryan> yeah, pretty soon
[8:30] <mikeryan> if you just want a stable version, i recommend using the apt-get install method
[8:30] <mikeryan> it will be the same version across all three nodes and it will take very little effort from you
[8:30] <kermin> but after some time when developers released new version I cant setup with my older version by using apt-get , right ?
[8:31] <mikeryan> there *might* be a way to hold a previous version, but i wouldn't rely on it
[8:33] <kermin> my all efforts are just to check whether ceph is intelligent enough to reduce traffic across osds based on application usage . if yes I'm done , if not find a way.
[8:34] <kermin> so wanted to have same version for all my testing work. than to make a good report on it. :)
[8:37] <kermin> CRUSH is really imp part of ceph . But if for certain computation I need data that is not available locally or within acceptable network latency . ceph should have some solution for that. may be dynamic changes to primary etc. at this point i am not sure even its feasible or not.
[8:40] <mikeryan> if you're awake during US west coast business hours, that's the best time to reach the people who can talk intelligently about that
[8:40] <mikeryan> alternatively, send email to ceph-devel@vger.kernel.org
[8:40] * grant (~grant@202-173-147-27.mach.com.au) Quit (Remote host closed the connection)
[8:41] <kermin> I generally post my queries to ceph-devel@vger.kernel.org.
[8:41] <kermin> yeah sure Sir.
[8:43] <kermin> Thank you for your help. :)
[8:49] * gaveen (~gaveen@ has joined #ceph
[8:54] * Ul (~Thunderbi@ has joined #ceph
[8:58] <Ul> hi everybody. I'm looking at building a small ceph cluster for hosting kvm virtual machines with roughly 10TB storage. currently looking at some server systems and was wondering if you could give me some advice whether this is the right choice. I'm looking at SuperMicro FatTwin systems or SuperMicro Microcloud. Would it be better to go for 4 nodes with 4 disks or 8 nodes with 2 disks? The 2 disk systems don't allow having a third storage device so having a SSD
[8:59] * pmjdebru1jn (~pmjdebrui@overlord.pcode.nl) Quit (Remote host closed the connection)
[9:01] <mikeryan> Ul: you probably won't find anyone awake for another several hours
[9:01] <mikeryan> the best place to direct that message is ceph-devel@vger.kernel.org
[9:02] <kermin> mikeryan : Sir, I am now able to execute "sudo dpkg-buildpackage " but inside cloned ceph from git
[9:03] * jlogan (~Thunderbi@ Quit (Quit: jlogan)
[9:04] <Fruit> how do I tell ceph to reopen its logfiles? service ceph reload doesn't seem to work
[9:06] <Ul> thanks
[9:09] * verwilst (~verwilst@d5152FEFB.static.telenet.be) has joined #ceph
[9:26] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has joined #ceph
[9:33] * synapsr (~synapsr@c-69-181-244-219.hsd1.ca.comcast.net) has joined #ceph
[9:41] * loicd (~loic@ has joined #ceph
[9:54] * synapsr (~synapsr@c-69-181-244-219.hsd1.ca.comcast.net) Quit (Remote host closed the connection)
[9:58] * The_Bishop (~bishop@2001:470:50b6:0:f58c:9d20:6367:a97b) Quit (Ping timeout: 480 seconds)
[10:00] * synapsr (~synapsr@c-69-181-244-219.hsd1.ca.comcast.net) has joined #ceph
[10:06] * synapsr (~synapsr@c-69-181-244-219.hsd1.ca.comcast.net) Quit (Remote host closed the connection)
[10:06] * The_Bishop (~bishop@2001:470:50b6:0:ed64:3fff:1bea:faa) has joined #ceph
[10:08] * renzhi (~renzhi@ Quit (Quit: Leaving)
[10:10] * miroslavk (~miroslavk@173-228-38-131.dsl.dynamic.sonic.net) Quit (Quit: Leaving.)
[10:20] * MikeMcClurg (~mike@firewall.ctxuk.citrix.com) has joined #ceph
[10:33] * miroslavk (~miroslavk@173-228-38-131.dsl.dynamic.sonic.net) has joined #ceph
[10:46] * miroslavk (~miroslavk@173-228-38-131.dsl.dynamic.sonic.net) Quit (Quit: Leaving.)
[10:51] * tziOm (~bjornar@ has joined #ceph
[10:52] <tziOm> I dont understand the chicken/egg problem with having to use 3 mons.. when I add mon#2, my cluster hangs!
[10:57] <joao> is that a second monitor?
[10:58] <joao> also, morning #ceph
[10:58] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[10:58] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[11:01] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[11:01] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[11:05] <tziOm> nah.. it was my mistake more or less (and a new bug in ceph)
[11:05] <tziOm> following the: http://ceph.com/docs/master/cluster-ops/add-or-rm-mons/
[11:06] <tziOm> problem is keyring is not copied to /var/lib/ceph/mon/ceph-X/keyring
[11:06] <tziOm> so ceph-mon crashes
[11:06] <tziOm> auth: error reading file: /var/lib/ceph/mon/ceph-c/keyring: can't open /var/lib/ceph/mon/ceph-c/keyring: (2) No such file or directory
[11:06] <tziOm> mon.c@-1(probing) e0 unable to load initial keyring /etc/ceph/ceph.keyring
[11:07] <tziOm> common/Thread.cc: In function 'int Thread::join(void**)' thread 7fa75c6af780 time 2012-10-11 11:03:31.318274
[11:07] <tziOm> common/Thread.cc: 117: FAILED assert("join on thread that was never started" == 0)
[11:07] <tziOm> CRASH
[11:07] <joao> oh, I see
[11:07] <joao> I wonder how that happens; probably during mkfs or something
[11:08] <tziOm> mkfs says something about not able to connect to admin socket.... another chicken/egg
[11:08] <joao> and I mean the failure in copying the keyring
[11:08] <tziOm> atleast I followed the guide to the letter
[11:09] * oliver1 (~oliver@jump.filoo.de) has joined #ceph
[11:09] <tziOm> ..and it should not crash anyway.. rather say no key found or whatever
[11:09] <joao> tziOm, that's probably some client trying to use the admin socket during mkfs; I've seen that before, but unless you are actually running the monitor, there's no reason for concern
[11:09] <joao> I believe not being able to connect to the admin socket might even be expected
[11:09] <joao> ceph is just too verbose
[11:10] <joao> tziOm, the monitor requires the keyring to work properly
[11:10] <tziOm> cant connect to admin socket of something you are trying to set up..
[11:10] <tziOm> joao, the keyring works
[11:10] <tziOm> so just copying keyring from /etc/ceph.keyring solved problem
[11:11] <joao> I see; I'm still waking up (just got out of bed)
[11:12] <tziOm> About speed of ceph.. I have bonded Gb interfaces giving me approx 1500mbit in netperf
[11:12] <joao> tziOm, my point with the mkfs outputing those messages is that, during mkfs, there is little to do with an admin socket, and the code path might not even allow you to register it anyway
[11:12] <tziOm> each osd gives me approx 220MB/sec write
[11:12] <joao> mkfs's job is only to set up the system
[11:12] <joao> and then exit
[11:13] <tziOm> I only get ~24MB/sec on rados bench -p pbench 900 write
[11:13] <tziOm> 4 osds
[11:14] <joao> tziOm, nhm put a nice blog post about performance on ceph.com; I'm not much into performance, so any questions you might have on that should probably taken to him ;)
[11:14] <tziOm> have a direct link to this?
[11:14] <tziOm> ah.. on the top!
[11:26] <joao> tziOm, just to confirm, are you using 0.48.2?
[11:35] <joao> tziOm, the next time you happen to bump into that keyring not being loaded issue, try running the --mkfs with --debug-mon 10 and check if it outputs some message about moving the key to a separate keyring
[11:40] <tziOm> 52.2
[11:50] <joao> tziOm, any chance you can share how your keyring file was generated and its contents?
[11:51] <joao> I might be on to something regarding why it was not copied during mkfs, but would need to make sure before crying victory
[11:59] <tziOm> yep..
[12:00] <Qten> hi all, anyone know of any notes/doco on howto configure cinder w/ceph?
[12:01] <tziOm> ceph auth get mon. > /tmp/mon.key
[12:01] <tziOm> ceph-mon -i c --mkfs --monmap /tmp/mon.map --keyring /tmp/mon.key
[12:01] <joao> have you opened that file?
[12:01] <joao> :p
[12:02] <tziOm> ah!
[12:02] <joao> I'm not sure what's wrong, but maybe the 'mon.' bit?
[12:02] <tziOm> one of the mons answer with that... for some reason
[12:02] <joao> try dropping the '.'
[12:02] <tziOm> that does not work
[12:03] <tziOm> mon. is correct...
[12:03] <joao> I suppose so
[12:03] <joao> not very familiar with that
[12:03] <Qten> nevermind i think i found what i was after
[12:03] <tziOm> what this means then is ceph should print something to stderr!!!!!
[12:03] <tziOm> it (swearword!) prints errors to stdout..!!
[12:04] <joao> so, you say one monitor replies with that 'failed to find mon.'? does the other one do too?
[12:04] <tziOm> wth is wrong with that? pretty basic stuff
[12:04] <tziOm> no, one monitor gives the key just fine
[12:04] <joao> tziOm, the 'ceph' tool has a peculiar way to deal with commands
[12:04] <tziOm> so random when I dont get it
[12:04] <joao> it appears that one monitor has a screwed up keyring
[12:05] <tziOm> yes... I dont understand the keyring stuff...
[12:05] <tziOm> they cant all share a key?
[12:05] <joao> they must share a key in order to work
[12:05] <joao> I mean, in order to talk to each other
[12:07] <joao> tziOm, could you please run 'ceph -m <monitor-ip:monitor-port> auth get mon.' for each monitor you have running?
[12:07] <tziOm> yeah..
[12:07] <joao> flush out the one with the wrong keyring
[12:07] <tziOm> joao, I have, and only first monitor has keyring, others dont for some reason (alltho they work just fine...)
[12:07] <joao> and then check the contents of that keyring under mon-data-dir/keyring?
[12:08] <joao> ah
[12:08] <tziOm> ah.. seems I have a keyring problem somewhere..
[12:11] * Cube1 (~Adium@cpe-76-95-223-199.socal.res.rr.com) has joined #ceph
[12:12] * elder (~elder@2607:f298:a:607:5500:a2f8:c48f:c5d4) Quit (Ping timeout: 480 seconds)
[12:14] <tziOm> How do i find locating of an object?
[12:21] * elder (~elder@2607:f298:a:607:1059:f4c1:babe:9559) has joined #ceph
[12:32] * Cube1 (~Adium@cpe-76-95-223-199.socal.res.rr.com) Quit (Quit: Leaving.)
[12:35] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[12:35] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[12:40] * brambles (xymox@grip.espace-win.org) has joined #ceph
[12:45] <oliver1> Hi Alex... uhm, do you @inktank get aware of someone - me - feeding resolved tickets with new updates?
[12:47] * BManojlovic (~steki@ has joined #ceph
[12:54] * loicd (~loic@ Quit (Ping timeout: 480 seconds)
[13:01] <tziOm> what is the problem when an osd is down but still in?
[13:02] <tziOm> for example if I stop ceph-osd ?
[13:04] <tziOm> should ceph osd create "uuid" work?
[13:04] <tziOm> seems it does not use the uuid passed..
[13:06] <tziOm> yes it does...
[13:13] <joao> oliver1, I did receive an email with the update from the tracker
[13:15] <oliver1> Yes, should have been two in the last two weeks... just wondering, if it's already covered with the TONS of patches ;)
[13:16] * Ul (~Thunderbi@ Quit (Ping timeout: 480 seconds)
[13:18] <oliver1> joao, this is: #2573 + #2075
[13:19] <joao> oliver1, no idea. elder probably knows better than I do :)
[13:20] <oliver1> yeah, should be )
[13:43] * deepsa (~deepsa@ Quit (Quit: Computer has gone to sleep.)
[13:47] * dty (~derek@pool-71-178-175-208.washdc.fios.verizon.net) has joined #ceph
[14:17] * lxo (~aoliva@lxo.user.oftc.net) Quit (Quit: later)
[14:18] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[14:20] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[14:20] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[14:31] * loicd (~loic@magenta.dachary.org) has joined #ceph
[14:37] * kermin (ca037809@ircip2.mibbit.com) Quit (Quit: http://www.mibbit.com ajax IRC Client)
[14:46] * dty (~derek@pool-71-178-175-208.washdc.fios.verizon.net) Quit (Quit: dty)
[14:48] * gaveen (~gaveen@ Quit (Ping timeout: 480 seconds)
[14:52] * Leseb (~Leseb@ has joined #ceph
[14:57] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[14:57] * loicd (~loic@ has joined #ceph
[14:58] * gaveen (~gaveen@ has joined #ceph
[15:16] * dty (~derek@testproxy.umiacs.umd.edu) has joined #ceph
[15:18] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[15:19] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[15:35] * lofejndif (~lsqavnbok@28IAAIAKT.tor-irc.dnsbl.oftc.net) has joined #ceph
[16:01] * lofejndif (~lsqavnbok@28IAAIAKT.tor-irc.dnsbl.oftc.net) Quit (Quit: gone)
[16:04] * alexxy (~alexxy@2001:470:1f14:106::2) has joined #ceph
[16:05] <alexxy> hi all!
[16:05] <alexxy> does binding to cephfs if osd is same host still may lead to deadlock?
[16:14] * ninkotech (~duplo@ has joined #ceph
[16:19] <tziOm> how do you guys think cephfs performs on mkdir -p {0..255}/{0..255}
[16:20] <tziOm> and also rm -r * in this folder after..
[16:22] <stan_theman> not ultra happy when i've run a similar type of command
[16:23] <tziOm> the remove takes _forever_
[16:23] <stan_theman> yeah
[16:23] <tziOm> laptop mkdir: real 0m5.230s
[16:24] <tziOm> laptop rm -r: real 0m1.261s
[16:26] <tziOm> cephfs mkdir (4 osd, 1 mds, 3 mon, 2Gbit): real 2m27.276s
[16:28] <tziOm> cephfs rm -r: forever
[16:29] <stan_theman> :D
[16:30] <tziOm> I know cephfs is not "production quality"... but really..
[16:31] <tziOm> what are they doing here?
[16:31] <tziOm> seems it tales approx 30 secs to remove 1 folder (with 256 subfolders)
[16:33] <tziOm> I will have to come back with the rm number.. its 170 folders left, and process started when I posted the mkdir for ceph
[16:34] <tziOm> later
[16:34] * tziOm (~bjornar@ Quit (Remote host closed the connection)
[16:51] * mengesb (~bmenges@servepath-gw3.servepath.com) Quit (Quit: Leaving.)
[16:55] * markl (~mark@tpsit.com) has joined #ceph
[16:56] * synapsr (~synapsr@c-69-181-244-219.hsd1.ca.comcast.net) has joined #ceph
[17:03] * PerlStalker (~PerlStalk@ has joined #ceph
[17:15] * verwilst (~verwilst@d5152FEFB.static.telenet.be) Quit (Quit: Ex-Chat)
[17:21] <via> is there an up-to-date method for compiling the ceph kernel module for 2.6.32 (rhel6)?
[17:22] <via> or is it considered no logner supported
[17:29] <nhm> via: 2.6.32 is pretty old at this point...
[17:29] <via> i mean, i know that
[17:29] <via> but its what el6 uses
[17:30] <nhm> via: It might work ok with all of the backported they do.
[17:30] <via> well, i couldn't get it to compile
[17:31] <via> which is why i'm asking
[17:32] <nhm> via: I imagine nothing here is helping? http://ceph.com/wiki/Installing_on_RedHat_or_CentOS
[17:33] <via> none of that has anything to do with the kernel modules
[17:33] <via> at least, as far as i can see. i built rpms and all
[17:33] <via> i found a bunch of random mailing list posts about reverting commits for workqueue support
[17:33] <via> and did so
[17:33] <via> but it still does not compile
[17:33] <via> hence asking if there was an up-to-date method, or if its just not supported anymore
[17:34] <nhm> via: hrm, the spec files don't build the kernel client?
[17:34] <nhm> via: anyway, I confess this is not my area. I think you already know more than I do about it. :/
[17:34] * tziOm (~bjornar@ti0099a340-dhcp0358.bb.online.no) has joined #ceph
[17:35] <via> nope, nothing in the specfile afaict
[17:35] <tziOm> Ok, back with results from rm -r ...
[17:35] <nhm> via: you might try asking Gary Lowell.
[17:35] <tziOm> real 38m25.685s
[17:35] <tziOm> that is mkdir -p {0..255}/{0..255} ; time rm -r *
[17:35] <tziOm> 38 minutes!!
[17:36] <tziOm> insane
[17:38] <nhm> tziOm: would you mind trying the fileop benchmark from iozone3?
[17:38] <tziOm> I could try.
[17:38] <tziOm> what params would you like?
[17:38] <nhm> tziOm: 1 byte files, 8000 files (-f 20)
[17:40] <nhm> it'll loop after it gets through all of the tests, so just ctrl+c it once it goes back and starts the first test over again.
[17:40] <nhm> That'll provide info about how many ops/s it's doing for different operation types.
[17:41] <tziOm> so just iozone -f20
[17:41] <tziOm> ?
[17:41] <nhm> "fileop -f 20" should do it.
[17:42] <tziOm> this is ceph-fuse
[17:42] <tziOm> cant even manage to mount with kernel module (3.6.1)
[17:42] * synapsr (~synapsr@c-69-181-244-219.hsd1.ca.comcast.net) Quit (Remote host closed the connection)
[17:43] <tziOm> I doubt it will ever finish (fileop)
[17:45] * vata (~vata@2607:fad8:4:0:8d6a:59a3:71b9:93f4) has joined #ceph
[17:52] * loicd (~loic@ Quit (Quit: Leaving.)
[17:53] * loicd (~loic@magenta.dachary.org) has joined #ceph
[17:53] * rweeks (~rweeks@c-98-234-186-68.hsd1.ca.comcast.net) has joined #ceph
[17:54] * oliver1 (~oliver@jump.filoo.de) has left #ceph
[18:08] * aliguori (~anthony@cpe-70-123-130-163.austin.res.rr.com) Quit (Quit: Ex-Chat)
[18:10] <tziOm> . mkdir rmdir create read write close stat access chmod readdir link unlink delete Total_files
[18:10] <tziOm> A 20 1319 197 613 1069 11305 52028 12687 12672 1258 19489 11 964 22 8000
[18:10] * nwatkins (~nwatkins@soenat3.cse.ucsc.edu) has joined #ceph
[18:11] <nhm> hrm, hard to read that with how it lined up in irc. looks like rmdir is slow, and link and delete are also quite slow?
[18:15] <joao> surely some contention on the mds?
[18:17] <tziOm> ?
[18:17] <tziOm> how can I debug?
[18:17] <Fruit> my experiences are similar: slow fileops and 99% cpu usage on the mds
[18:19] <tziOm> not much load on mds.. ~5% cpu load
[18:19] <nhm> tziOm: Might be lock contention or something.
[18:22] <nhm> tziOm: your readdir performance is actually quite a bit better than on a test I did a little while back, but your link and delete are worse.
[18:23] * sjusthm (~sam@66-214-139-112.dhcp.gldl.ca.charter.com) has joined #ceph
[18:23] <nhm> your rmdir performance is worse but your create performnace is better.
[18:23] <tziOm> this can not be called "performance"
[18:24] <Karcaw> quit
[18:24] * Karcaw (~evan@68-186-68-219.dhcp.knwc.wa.charter.com) Quit (Quit: leaving)
[18:26] <gregaf> via: I don't think the kernel module ever built against 2.6.32 — it got merged for 2.6.34 and has seen quite a lot of change since then
[18:26] <nhm> tziOm: cephfs has bugs, we've been very open about that. While several of the tests were very slow in the results you linked, others were quite fast.
[18:27] <gregaf> unfortunately we don't have the resources yet to do backports like that; elder is starting to do some now but that's for like 3.4 or 3.5 ;)
[18:28] <gregaf> tziOm: Fruit: nhm: rm currently requires a synchronous disk writes for each file, which even on a very fast storage system is going to be a lot slower than you're used to
[18:28] <gregaf> if rados bench reports 20MB/s then it's going to be...atrocious
[18:28] <gregaf> this is a known weak spot in our metadata handling and will be one of the first performance things we go after, but for the moment you have to live with it (or move on) :(
[18:29] <gregaf> I don't remember the whole story with mkdir; there's probably room for improvement there in most circumstances, but that part is competitive with other distributed FSes IIRC
[18:29] <elder> gregaf is right. See https://github.com/ceph/ceph-client/tree/linux-3.5.5-ceph and https://github.com/ceph/ceph-client/tree/linux-3.4.12-ceph
[18:30] <elder> But updates to that are occasional.
[18:31] <elder> No more earlier than that, and there may not be unless there becomes a strong business reason to do so.
[18:31] <gregaf> s/a strong/a dominatingly strong
[18:31] <elder> For some definition of "dominatingly"
[18:32] <nhm> gregaf: do you remember the difference between delete and unlink?
[18:32] <tziOm> what does atrocious mean?
[18:32] <nhm> gregaf: I'm curious why unlink seems to be significantly faster.
[18:32] <elder> tziOm, bad, very very bad.
[18:33] <tziOm> rados bench reports ~40MB/s write here
[18:33] <tziOm> on rados bench -p pbench 900 write
[18:33] <gregaf> *ponders*
[18:34] <tziOm> dont know how that is.. but raidsets on each of 4 osds write approx 220MB/sec...
[18:34] <gregaf> nhm: if I remember the distinction correctly (which is no guarantee) then unlink can be represented in MDS memory and flushed out to disk asynchronously since it's just a namespace update and if both the client and the MDS crash you're allowed to lose it (assuming nobody else has seen the visible update)
[18:35] <gregaf> but the delete requires that you log that the inode is gone to make sure nobody else has extra references to it hanging around
[18:35] <nhm> gregaf: interesting, that would explain it.
[18:35] <gregaf> that's a bit fuzzy and I don't remember for certain how much of it is a real barrier versus just being lazy about the implementation
[18:36] <gregaf> nhm: oh wait, what's your distinction on delete versus unlink here?
[18:36] <gregaf> directory delete and file unlink, maybe?
[18:36] <elder> Is a rados pool id 32 bits or 64 bits?
[18:36] * gaveen (~gaveen@ Quit (Remote host closed the connection)
[18:37] <gregaf> elder: I think I remember it being…both, due to some bad types in various important places
[18:37] <elder> That's what it looks like to me...
[18:37] <elder> Thought I'd make sure I wasn't crazy. A layout defines it as 32 buits.
[18:37] * slang (~slang@ace.ops.newdream.net) has left #ceph
[18:37] <elder> (French for "bits"
[18:38] * slang (~slang@ace.ops.newdream.net) has joined #ceph
[18:38] <gregaf> luckily nobody yet has a cluster where >4billion pools is feasible
[18:38] <elder> (Yet)
[18:38] <tziOm> hmm..
[18:39] <nhm> gregaf: http://www.iozone.org/src/current/fileop.c
[18:39] <nhm> gregaf: see file_unlink and file_delete
[18:41] <nhm> note: I haven't looked at this closely yet.
[18:41] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) Quit (Quit: Leaving.)
[18:42] <gregaf> nhm: without digging into it, it does appear to be that the unlink is removing extra links whereas the delete is removing the last link
[18:42] <gregaf> so, what I said was probably right even though it hurt my head
[18:42] <mikeryan> there was a guy last night with this question
[18:42] <mikeryan> Hey guys, I'd like to include the following btrfs options when
[18:42] <mikeryan> using mkcephfs - can you please help me with where to specify?
[18:42] <mikeryan> mkfs.btfs options: -l 16k -n 16k
[18:42] <mikeryan> he got that from nhm's performance blog post
[18:43] <gregaf> mikeryan: that probably means he's using the btrfs devs options, and he shouldn't
[18:43] <nhm> mikeryan: I replied to him about it... I'm not sure if you can do that with mkcephfs. I just do it with mkfs.btrfs
[18:43] <gregaf> instead just format and mount the drives himself
[18:43] <gregaf> iirc
[18:55] * MikeMcClurg (~mike@firewall.ctxuk.citrix.com) Quit (Quit: Leaving.)
[18:56] * adjohn (~adjohn@108-225-130-229.lightspeed.sntcca.sbcglobal.net) has joined #ceph
[18:58] <tziOm> why cant I map a rbd
[18:58] <tziOm> it just hangs
[18:58] <tziOm> it worked yesterday...
[19:01] <sjusthm> tziOm: ceph -s output?
[19:03] * chutzpah (~chutz@ has joined #ceph
[19:04] <tziOm> health HEALTH_OK
[19:04] <tziOm> monmap e3: 3 mons at {a=,b=,c=}, election epoch 10, quorum 0,1,2 a,b,c
[19:04] <tziOm> osdmap e128: 6 osds: 4 up, 4 in
[19:04] <tziOm> pgmap v7043: 1352 pgs: 1262 active+clean, 90 active+clean+scrubbing; 40507 MB data, 86400 MB used, 1626 GB / 1787 GB avail
[19:04] <tziOm> mdsmap e169: 2/2/2 up {0=a=up:active,1=c=up:reconnect}
[19:05] <tziOm> now 2nd mds is active as well
[19:09] <tziOm> root@storage02:/# rbd map foo
[19:09] <tziOm> add failed: (5) Input/output error
[19:14] <joao> going to be late for the stand-up; the router is acting funny again and I need to reboot it in order to get the laptop to access the internet -_-
[19:15] * Cube1 (~Adium@cpe-76-95-223-199.socal.res.rr.com) has joined #ceph
[19:17] * jluis (~JL@ has joined #ceph
[19:18] * synapsr (~synapsr@c-69-181-244-219.hsd1.ca.comcast.net) has joined #ceph
[19:21] * The_Bishop (~bishop@2001:470:50b6:0:ed64:3fff:1bea:faa) Quit (Ping timeout: 480 seconds)
[19:22] * joao (~JL@ Quit (Ping timeout: 480 seconds)
[19:23] * jluis is now known as joao
[19:24] <tziOm> how can I debug rbd
[19:24] <tziOm> I can create and list and everything, but mapping fails
[19:30] * The_Bishop (~bishop@2001:470:50b6:0:719e:9c41:8a74:8843) has joined #ceph
[19:32] * synapsr (~synapsr@c-69-181-244-219.hsd1.ca.comcast.net) Quit (Remote host closed the connection)
[19:34] * Leseb (~Leseb@ Quit (Quit: Leseb)
[19:34] * maelfius (~mdrnstm@ has joined #ceph
[19:35] * Ryan_Lane (~Adium@c-67-160-217-184.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[19:35] * davidz (~Adium@ip68-96-75-123.oc.oc.cox.net) has joined #ceph
[19:36] * nhm_ (~nhm@174-20-101-163.mpls.qwest.net) has joined #ceph
[19:40] * nhm (~nhm@174-20-35-45.mpls.qwest.net) Quit (Ping timeout: 480 seconds)
[19:41] * miroslavk (~miroslavk@c-98-234-186-68.hsd1.ca.comcast.net) has joined #ceph
[19:46] * joshd (~joshd@2607:f298:a:607:221:70ff:fe33:3fe3) has joined #ceph
[19:50] <via> i have a cluster with two nodes, one being osd+mds+mon and the other being just osd... and i have 30 pg's that are active+degraded. querying those pg's shows nothing wrong except there is only 1 of them
[19:50] <joshd> Qten: http://ceph.com/docs/wip-rbd-openstack-doc/rbd/rbd-openstack/
[19:50] <via> and not 2
[19:50] <via> but i cannot get it to replicate it and stop being degraded
[19:52] * deepsa (~deepsa@ has joined #ceph
[19:52] <via> http://pastebin.com/b7bpCJKD -- the replication 'size' of each pool is 2
[19:53] <via> and i've tried scrubbing and repairing both osd's
[19:56] * Leseb (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) has joined #ceph
[19:57] * Leseb_ (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) has joined #ceph
[19:57] * Leseb (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) Quit (Read error: Connection reset by peer)
[19:57] * Leseb_ is now known as Leseb
[19:59] * jstrunk (~quassel@ Quit (Remote host closed the connection)
[19:59] <joshd> via: the problem is that those pgs are only getting mapped to one osd
[19:59] <joshd> via: what's your crushmap?
[20:00] * adjohn (~adjohn@108-225-130-229.lightspeed.sntcca.sbcglobal.net) Quit (Quit: adjohn)
[20:00] * jstrunk (~quassel@ has joined #ceph
[20:00] <tziOm> joshd, Any idea why I am not able to map rbd's? how to debug this?
[20:01] <joshd> tziOm: anything in dmesg?
[20:01] <via> joshd: as in i should decompile and paste it? or just ceph osd tree?
[20:01] <joshd> via: decompile
[20:01] <tziOm> yeah
[20:01] <tziOm> lots of stuff
[20:01] <tziOm> front:
[20:02] * steki-BLAH (~steki@ has joined #ceph
[20:02] <via> joshd: http://pastebin.com/kSiVJFuP
[20:04] <joshd> via: is osd.2 on a separate host?
[20:05] <joshd> via: it's strange to have it in a different level of the hierarchy than other osds like that
[20:05] <joshd> via: it may cause you to hit a bug with the default crushmap
[20:05] <via> yeah, i know its a bit weird, but its because i added it later
[20:05] <via> but no, its a second host
[20:06] <via> i can add it to the same 'rack'
[20:06] <tziOm> joshd, libceph: osdc handle_map corrupt msg
[20:07] <joshd> tziOm: that would be the problem - did you change your crushmap (maybe used a 'tree' alg instead of 'straw')?
[20:07] * deepsa (~deepsa@ Quit (Quit: Computer has gone to sleep.)
[20:07] <joshd> via: you can get rid of the rack layer entirely, and just have two hosts, each with one osd
[20:08] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[20:08] <via> yep, i'm trying now
[20:08] * loicd (~loic@magenta.dachary.org) has joined #ceph
[20:09] <via> okay, its set and i'm repairing, i'll see what happens. thank you
[20:09] <joshd> you're welcome
[20:10] <joshd> tziOm: the kernel client can't handle some newer features of crush still
[20:10] <tziOm> ok..
[20:10] <tziOm> patches are where?
[20:11] <joshd> tziOm: linux 3.5 has the crush tunables support
[20:12] <joshd> tziOm: other algorithms don't matter as much (straw is more than sufficient everywhere), so that's simply not implemented in the kernel yet (http://www.tracker.newdream.net/issues/2759)
[20:13] <tziOm> but this means I cant use rbd with 3.6?
[20:13] <joshd> no, it just means you can't use e.g. the tree algorithm with the kernel client
[20:14] <tziOm> hmm..
[20:14] <tziOm> Does it explain why I cant rbd map foo?
[20:14] <joshd> can you paste your current crushmap?
[20:15] <tziOm> http://pastie.org/5036576
[20:18] <joshd> I don't see anything unsupported by the kernel there
[20:18] <joshd> are you using kernel 3.6?
[20:18] * Karcaw (~evan@68-186-68-219.dhcp.knwc.wa.charter.com) has joined #ceph
[20:19] <joshd> did you enable crush tunables?
[20:21] <tziOm> dont know.. lets see
[20:22] <tziOm> where should that be located? cant find it in kernel config..
[20:23] <tziOm> 3.6.1 to be specific
[20:24] <joshd> it's a property of the crushmap, not the kernel. 3.6 should support them just fine
[20:24] <tziOm> did you not see my crushmap
[20:25] <tziOm> so this is a ceph configure setting, you mean?
[20:25] * glowell1 (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) has joined #ceph
[20:26] * glowell (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) Quit (Read error: No route to host)
[20:26] <joshd> I'm not sure if it's included in the decompiled crush output
[20:26] <joshd> I'm talking about these: http://ceph.com/docs/wip-rbd-openstack-doc/cluster-ops/crush-map/#tunables
[20:26] <via> joshd: so, the repair completed, but now there are a bunch of pg's in the 'active+remapped+replay' state as well as 30+ pg's in 'active+remapped' state
[20:27] <tziOm> I have not set that, no
[20:28] * nhorman (~nhorman@hmsreliant.think-freely.org) has joined #ceph
[20:28] <joshd> via: the replay state should go away in 30s iirc
[20:29] <tziOm> joshd, should I set tunables?
[20:30] <joshd> tziOm: no, that's not what's preventing the kernel client from interpreting the crushmap
[20:31] * synapsr (~synapsr@c-98-234-186-68.hsd1.ca.comcast.net) has joined #ceph
[20:31] <joshd> tziOm: I'm not sure what is, but if you can figure out what property of your crushmap is making 'rbd map' fail, we can fix the bug
[20:31] <via> joshd: they've been there for 5 minutes, not changing
[20:31] <via> does the remapped ever go away?
[20:31] <joshd> via: remapped normally goes away when recovery is complete
[20:32] <joshd> via: which version are you running?
[20:32] * Ryan_Lane (~Adium@ has joined #ceph
[20:32] <via> .48 argonaut
[20:32] <tziOm> joshd, how can I figure?
[20:34] <joshd> tziOm: you had it working before, with a different crushmap, right? compare those and figure out what changed to make rbd map fail, i.e. you could change the crushmap back to the working version, and change one bit at a time until rbd map fails
[20:35] <tziOm> hmm..
[20:36] <joshd> via: what's ceph pg dump?
[20:36] <joshd> via: and your new crushmap (decompiled) would be good to check too
[20:37] <via> there are a lot of pgs....hold on
[20:39] <via> http://pastebin.com/pPbgY7c1 http://pastebin.com/25ba8cUv
[20:39] <tziOm> joshd, what exactly should I change in the crushmap you think?
[20:41] <joshd> tziOm: I'm not sure, but maybe try eliminating the extra level (datacenter) in the hierarchy
[20:43] <joshd> via: it looks like they're staying remapped because you're hitting the same problem tziOm did yesterday with the choose rules and only having 2 osds
[20:43] * glowell (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) has joined #ceph
[20:43] <via> i see
[20:44] <via> i don't suppose that was resolved <_<
[20:44] * glowell1 (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) Quit (Read error: No route to host)
[20:44] <joshd> via: it seems like there are some other things in his setup
[20:45] <joshd> via: but if you change choose to chooseleaf in the crush rules, it should never only map to 1 osd
[20:46] <joshd> I'll be back in a bit
[20:46] <via> would choose firstn 2 also work? i'm trying to understand what this means
[20:46] <joao> do we have a c-only component being compiled on the ceph tree?
[20:46] <joao> any ideas?
[20:46] <joshd> joao: the mount helpers
[20:47] <joao> thanks
[20:47] <joshd> via: no, it's a problem with choose not going down to a leaf node, which chooseleaf does
[20:48] <via> okay
[20:48] <joao> and anyone has any idea how to go about specifying that those sources should be compiled with gcc instead of g++?
[20:48] <tziOm> removed datacenter, no change..
[20:48] <joao> the Makefile.am is not much help
[20:49] <joshd> via: also take a look at http://ceph.com/docs/wip-rbd-openstack-doc/cluster-ops/crush-map/#tunables
[20:51] <via> joshd: okay ..although those won't be supported before linux 3.5 i see. anyway, i changed it to chooseleaf and will wait to see what happens
[20:52] * aliguori (~anthony@ has joined #ceph
[20:53] * sjusthm (~sam@66-214-139-112.dhcp.gldl.ca.charter.com) Quit (Remote host closed the connection)
[20:53] <tziOm> joshd, any other idea?
[20:53] * sjusthm (~sam@66-214-139-112.dhcp.gldl.ca.charter.com) has joined #ceph
[20:58] <via> joshd: unfortunetely, it didn't change anything
[21:00] * The_Bishop (~bishop@2001:470:50b6:0:719e:9c41:8a74:8843) Quit (Ping timeout: 480 seconds)
[21:00] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[21:00] * loicd (~loic@magenta.dachary.org) has joined #ceph
[21:04] <via> joshd: i changed the crush map to eliminate the 'host' level completely and the pool just directly refers to the osd's
[21:04] <via> and that appears to have fixed it
[21:04] <via> but does this mean if i decide to make a cluster with 6 machines of two osd's each that i'll probably run into problems?
[21:05] <joshd> tziOm: I'm not sure what your old (working with rbd map) crushmap was
[21:06] <tziOm> it was the default
[21:07] <joshd> via: yeah, you may hit the same issue - you can test with the crushtool though
[21:08] <joshd> via: if you can avoid the kernel clients until 3.5, the tunables are the best solution
[21:09] <tziOm> ok..
[21:09] <tziOm> I think I solved now
[21:09] * The_Bishop (~bishop@2001:470:50b6:0:d9a0:fb9b:2d:3eac) has joined #ceph
[21:09] <tziOm> I did
[21:10] <tziOm> I just deleted 3 pools, test, swimmingpool and pbench
[21:11] <joshd> tziOm: deleting those pools fixed your problem?
[21:11] * MikeMcClurg (~mike@cpc10-cmbg15-2-0-cust205.5-4.cable.virginmedia.com) has joined #ceph
[21:12] * synapsr (~synapsr@c-98-234-186-68.hsd1.ca.comcast.net) Quit (Remote host closed the connection)
[21:12] <tziOm> yes
[21:13] <joshd> tziOm: was there anything special about those pools, or how they were created?
[21:13] <tziOm> no
[21:13] <tziOm> dont think so
[21:13] <tziOm> no.
[21:13] <tziOm> atleast I was able to map
[21:14] <tziOm> but seems kernel crashed when abusing the bd a little
[21:14] <tziOm> time dd if=/dev/zero of=foo bs=1024k count=1024
[21:14] <tziOm> dd: writing `foo': No space left on device
[21:14] <tziOm> 280+0 records in
[21:14] <tziOm> 279+0 records out
[21:14] <tziOm> 292945920 bytes (293 MB) copied, 0.606569 s, 483 MB/s
[21:14] <tziOm> real 0m0.625s
[21:14] <tziOm> user 0m0.000s
[21:15] <tziOm> sys 0m0.428s
[21:15] <tziOm> ...then no response from it..
[21:15] <tziOm> cant even ping it, so think its dead
[21:15] <tziOm> ...ceph has not figured yet, tho .. (1 minute)
[21:15] <tziOm> ok, there it went down, but not out
[21:16] <tziOm> How long time does it take from its down until out?
[21:17] <joshd> 5 minutes by default, to allow for transient failures
[21:18] <joshd> that's the 'mon osd down out internal'
[21:20] <tziOm> why is it only down, not out?
[21:21] <joshd> so data doesn't get reshuffled if the osd will be back soon (like when you restart a server)
[21:24] <via> so, i'm playing around with taking a node down to test availability... and even though it was perfectly healthy when i took down a node, ceph-fuse completely hung the mountpoint
[21:24] <via> is this supposed to happen?
[21:25] <joshd> via: with only two osds, yes
[21:25] * atrius (~atrius@24-179-64-97.dhcp.jcsn.tn.charter.com) has joined #ceph
[21:26] <joshd> via: with only 2 osds, the default heartbeat settings will make the failed osd not marked down until 1 minute has passed
[21:26] <joshd> via: so any writes going to that osd will be blocked until that happens
[21:26] <via> well, its down, and its been over 10 minutes
[21:27] * lofejndif (~lsqavnbok@9YYAAJRS2.tor-irc.dnsbl.oftc.net) has joined #ceph
[21:28] <joshd> via: in that case I'd guess it's a bug in cephfs
[21:29] <atrius> from the perspective of using rbd for backing storage of VM images and such... is there a reason i'd want 0.52 vice 0.48?
[21:29] <via> yeah... iwish i had the kernel module to see if it behaves similarly
[21:29] * nolan (~nolan@2001:470:1:41:20c:29ff:fe9a:60be) Quit (Quit: ZNC - http://znc.sourceforge.net)
[21:30] * The_Bishop (~bishop@2001:470:50b6:0:d9a0:fb9b:2d:3eac) Quit (Quit: Wer zum Teufel ist dieser Peer? Wenn ich den erwische dann werde ich ihm mal die Verbindung resetten!)
[21:30] <joshd> atrius: if you'd like to take advantage of rbd cloning, that's first released in 0.52
[21:31] * nolan (~nolan@2001:470:1:41:20c:29ff:fe9a:60be) has joined #ceph
[21:31] <atrius> joshd: which does look interesting.. but at this point i'll be all kinds of happy with "works and performs well" :D
[21:32] <via> yeah...plus i tried mounting with fuse on the other node and it worked for a minute and then hung completely... :-(
[21:32] <joshd> atrius: 0.48.2 is fine for that
[21:32] <atrius> joshd: okay :)
[21:32] <atrius> i'll look into other features after i have a basic setup alive and working well
[21:33] <atrius> at the moment, i only have one node available to do the job... while knowing that isn't ideal, is there any reason it wouldn't work?
[21:33] <joshd> atrius: if you can use qemu, the rbd caching will help a lot
[21:34] <atrius> joshd: you mean use qemu for the virtualization layer?
[21:34] <joshd> atrius: yes
[21:34] <joshd> since it's going directly through the userspace librbd
[21:34] <atrius> joshd: i'm going to be using KVM.. so yeah :)
[21:34] <joshd> atrius: great, one node won't be a problem either then
[21:34] <atrius> awesome :)
[21:36] <joshd> via: definitely something strange going on there... if you could capture logs with --debug-client 20 and --debug-ms 1 for ceph-fuse and put them in a bug report, that'd be great
[21:37] <dty> does someone know radosgw will only ever show you the buckets you created? I have given control FULL_CONTROL on the bucket through boto and confirmed that the user can access the bucket. Still when I request all the buckets for the user I do not see it listed.
[21:38] <gregaf> dty: yehudasa would know for sure, but I don't believe the S3 protocol is supposed to work like that, so it's doing exactly is it should :)
[21:38] <atrius> anything particularly special i should know about ceph+openstack?
[21:39] <joshd> atrius: http://ceph.com/docs/master/rbd/rbd-openstack/
[21:39] <yehudasa> dty: you can only see the buckets that you're the owner of, not the buckets that you have full control
[21:41] <atrius> joshd: thanks :)
[21:43] <dty> ok, well that makes my plans a little less saucy :) so the only way a user knows about another users bucket is via out of band?
[21:45] <yehudasa> dty: yes, unless you want to modify the radosgw code and add some functionality, but it's not out of the box
[21:50] <via> joshd: okay, i'll be able to tomorrow morning
[21:51] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has joined #ceph
[21:55] <tziOm> joshd, isnt it strange that rbd crashed because of a few pools..?
[21:55] <joshd> tziOm: yes, very strange
[21:56] <joshd> tziOm: do you still have the dmesg output from when the map was failing?
[21:58] <tziOm> sure
[21:59] <tziOm> http://pastie.org/5037010
[21:59] * tryggvil (~tryggvil@s529d22d5.adsl.online.nl) has joined #ceph
[22:00] <tziOm> this is freakin unstable
[22:01] <tziOm> I do a " dd if=/dev/zero of=foo bs=1024k count=512" on a rbd
[22:01] <tziOm> and if hangs totally.. cant even kill dd
[22:02] <tziOm> here is the kernel crash
[22:02] <tziOm> http://pastie.org/5037022
[22:03] <joshd> tziOm: that's btrfs on top of rbd, correct?
[22:03] <tziOm> yes
[22:04] * nwatkins (~nwatkins@soenat3.cse.ucsc.edu) Quit (Remote host closed the connection)
[22:04] <tziOm> atleast it is btrfs on rbd
[22:04] <tziOm> on osd as well..
[22:04] <tziOm> and mounting this on osd, so could be either
[22:06] * synapsr (~synapsr@c-98-234-186-68.hsd1.ca.comcast.net) has joined #ceph
[22:07] * The_Bishop (~bishop@2001:470:50b6:0:d9a0:fb9b:2d:3eac) has joined #ceph
[22:07] * dmick (~dmick@2607:f298:a:607:84b4:cd46:ebad:f64e) has joined #ceph
[22:08] * sjusthm (~sam@66-214-139-112.dhcp.gldl.ca.charter.com) Quit (Remote host closed the connection)
[22:15] <tziOm> tried with a ext4 formatted rbd, no change
[22:15] <tziOm> ..yes perhaps
[22:19] <joshd> tziOm: is this just a vanilla 3.6.1, or is there anything custom about it?
[22:19] <tziOm> vanilla
[22:19] * The_Bishop (~bishop@2001:470:50b6:0:d9a0:fb9b:2d:3eac) Quit (Ping timeout: 480 seconds)
[22:26] * via (~root@ Quit (Quit: leaving)
[22:27] * The_Bishop (~bishop@2001:470:50b6:0:719e:9c41:8a74:8843) has joined #ceph
[22:29] * nhorman (~nhorman@hmsreliant.think-freely.org) Quit (Quit: Leaving)
[22:39] * synapsr (~synapsr@c-98-234-186-68.hsd1.ca.comcast.net) Quit (Remote host closed the connection)
[22:42] * dty (~derek@testproxy.umiacs.umd.edu) Quit (Ping timeout: 480 seconds)
[22:43] * synapsr (~synapsr@c-98-234-186-68.hsd1.ca.comcast.net) has joined #ceph
[22:43] * adjohn (~adjohn@108-225-130-229.lightspeed.sntcca.sbcglobal.net) has joined #ceph
[22:46] <tziOm> I am not used to projects that "state" a certain level of maturity beeing this unstable!
[22:46] * tziOm (~bjornar@ti0099a340-dhcp0358.bb.online.no) Quit (Quit: Leaving)
[22:47] <rweeks> hm
[23:05] * MarkN (~nathan@ has joined #ceph
[23:05] * MarkN (~nathan@ has left #ceph
[23:16] <iggy> I have the sneaking suspicion that his problem was something local since a ton of people seem to be using rbd without any problems
[23:18] <joshd> yeah, he seems to be running into some strange things that no one else has
[23:19] <gregaf> it's a lot easier to get weird behavior when you futz around with CRUSH maps and don't know what you're doing
[23:19] <gregaf> that's one thing I know he did; there were others too
[23:19] * scalability-junk (~stp@188-193-208-44-dynip.superkabel.de) has joined #ceph
[23:20] * aliguori (~anthony@ Quit (Remote host closed the connection)
[23:30] * Karcaw (~evan@68-186-68-219.dhcp.knwc.wa.charter.com) Quit (Remote host closed the connection)
[23:32] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[23:32] * loicd (~loic@magenta.dachary.org) has joined #ceph
[23:35] * synapsr (~synapsr@c-98-234-186-68.hsd1.ca.comcast.net) Quit (Remote host closed the connection)
[23:41] * synapsr (~synapsr@c-98-234-186-68.hsd1.ca.comcast.net) has joined #ceph
[23:42] * synapsr (~synapsr@c-98-234-186-68.hsd1.ca.comcast.net) Quit (Remote host closed the connection)
[23:44] * rweeks (~rweeks@c-98-234-186-68.hsd1.ca.comcast.net) has left #ceph
[23:45] * lofejndif (~lsqavnbok@9YYAAJRS2.tor-irc.dnsbl.oftc.net) Quit (Quit: gone)
[23:52] * miroslavk (~miroslavk@c-98-234-186-68.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[23:57] * Karcaw (~evan@68-186-68-219.dhcp.knwc.wa.charter.com) has joined #ceph

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.