#ceph IRC Log


IRC Log for 2012-11-30

Timestamps are in GMT/BST.

[0:00] <phantomcircuit> hmm im using qemu-kvm with rbd
[0:01] <phantomcircuit> as soon as debian tries to access the disk it pegs the cpu
[0:01] <phantomcircuit> doesn't appear to actually be accessing anything
[0:01] <phantomcircuit> identical config with local disks works so it's something to do with rbd
[0:06] <phantomcircuit> huh gentoo works
[0:06] <phantomcircuit> debian being messed up :|
[0:15] <Robe> nhm: mersenne twisting all the iops?
[0:16] <sjustlaptop> nwatkins: you can extract them from the ceph_context
[0:16] <sjustlaptop> grep around for perf_counters
[0:18] * davidz (~Adium@ip68-96-75-123.oc.oc.cox.net) has joined #ceph
[0:37] * aliguori (~anthony@cpe-70-123-145-75.austin.res.rr.com) has joined #ceph
[0:41] * iggy2 (~iggy@theiggy.com) has joined #ceph
[0:44] <phantomcircuit> osd.0 [WRN] slow request 87.734037 seconds old
[0:44] <phantomcircuit> oops
[0:46] <lurbs> You can check which operation(s) were slow with: ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight
[0:47] <lurbs> Assuming standard socket locations, that is. Has to be on the machine running osd.0.
[0:51] <sagewk> nwatkins: you can't
[0:51] * iggy2 is now known as iggy_
[0:52] <lurbs> phantomcircuit: I'm aware of a bug that may be related to scrubbing that shows up like that. Don't suppose you have debugging turned way up?
[0:52] <phantomcircuit> no i have it at default
[0:55] <phantomcircuit> it's only the one osd
[0:55] <lurbs> If it's the one I think it is then restarting the OSD should clear it.
[0:55] <phantomcircuit> something is off with that one server
[0:55] * Enigmagic (enigmo@c-50-148-128-194.hsd1.ca.comcast.net) has joined #ceph
[0:56] <phantomcircuit> currently my "cluster" is only two physical systems
[0:56] <Enigmagic> is there any way to flush the ceph-fuse cache? i have a couple boxes where one thinks a file exists and the other thinks it doesn't. if i remount the fuse filesystem it will show up.
[0:56] <phantomcircuit> 4 hdds, 4 ssds
[0:56] <phantomcircuit> unfortuantely on the osd.0 box the ssds have max write throughput of only 40 MB/s each
[0:59] * ircolle (~ian@c-67-172-132-164.hsd1.co.comcast.net) Quit (Quit: ircolle)
[1:00] * aliguori (~anthony@cpe-70-123-145-75.austin.res.rr.com) Quit (Remote host closed the connection)
[1:03] * tnt (~tnt@83.164-67-87.adsl-dyn.isp.belgacom.be) Quit (Ping timeout: 480 seconds)
[1:03] <phantomcircuit> possibly i should just remove the osd on this system :|
[1:05] <phantomcircuit> actually i think i should read more about crush maps
[1:27] * Steki (~steki@gprswap.mts.telekom.rs) Quit (Ping timeout: 480 seconds)
[1:56] <nwatkins> sagewk: so are those counters in the objector logged some where?
[1:56] <sagewk> not logged
[1:56] <sagewk> you can pull them from the admin socket
[1:57] <nwatkins> does that admin socket only exist while the client instance is around?
[1:57] <sagewk> you can enable that from libcephfs, fwiw.. libceph_conf_set("admin_socket", "path....");
[1:57] <sagewk> yeah
[1:59] <nwatkins> sagewk: cool. so for the replica read stuff, do you think it makes more sense trying to interrogate the new read-from-replica counters in the unit test somehow, or after the fact from an admin socket dump?
[2:00] <sagewk> i think from the unit test...
[2:00] <sagewk> set teh socket to /tmp/something, do some localized reads, then query the socket and see if they were indeed localized
[2:01] <nwatkins> sagewk: ok great that makes sense.
[2:01] <sagewk> np
[2:08] * ChanServ sets mode +o joao
[2:08] * rweeks (~rweeks@c-24-4-66-108.hsd1.ca.comcast.net) Quit (Quit: ["Textual IRC Client: www.textualapp.com"])
[2:15] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[2:16] * loicd (~loic@magenta.dachary.org) has joined #ceph
[2:24] * davidz (~Adium@ip68-96-75-123.oc.oc.cox.net) Quit (Remote host closed the connection)
[2:43] * dmick (~dmick@2607:f298:a:607:7596:c46:d062:a53d) Quit (Quit: Leaving.)
[2:53] * adjohn (~adjohn@ Quit (Quit: adjohn)
[2:53] * adjohn (~adjohn@ has joined #ceph
[2:58] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[2:58] * loicd (~loic@magenta.dachary.org) has joined #ceph
[3:07] * adjohn (~adjohn@ Quit (Quit: adjohn)
[3:11] <phantomcircuit> if im getting 35 MB/s for a single rbd volume which has been mapped to /dev/rbd1 is it likely that that is the total throughput of the cluster?
[3:35] * jlogan1 (~Thunderbi@2600:c00:3010:1:e12a:776f:2a6d:8a8) Quit (Ping timeout: 480 seconds)
[3:36] * Ryan_Lane (~Adium@ Quit (Quit: Leaving.)
[3:40] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[3:40] * loicd (~loic@magenta.dachary.org) has joined #ceph
[3:52] * deepsa (~deepsa@ has joined #ceph
[4:35] * deepsa (~deepsa@ Quit (Quit: ["Textual IRC Client: www.textualapp.com"])
[4:36] * deepsa (~deepsa@ has joined #ceph
[4:46] * chutzpah (~chutz@ Quit (Quit: Leaving)
[5:01] * yanzheng (~zhyan@jfdmzpr05-ext.jf.intel.com) has joined #ceph
[5:01] * nwatkins (~nwatkins@c-50-131-197-174.hsd1.ca.comcast.net) Quit (Quit: Ex-Chat)
[5:07] * yanzheng (~zhyan@jfdmzpr05-ext.jf.intel.com) Quit (Quit: Leaving)
[5:50] * yoshi (~yoshi@p4105-ipngn4301marunouchi.tokyo.ocn.ne.jp) has joined #ceph
[6:32] <phantomcircuit> # ceph tell osd.2 bench
[6:32] <phantomcircuit> bench: wrote 1024 MB in blocks of 4096 KB in 8.750357 sec at 117 MB/sec
[6:32] <phantomcircuit> rbd devices seem to max out at 35MB/s though
[6:38] * yanzheng (~zhyan@jfdmzpr05-ext.jf.intel.com) has joined #ceph
[6:55] <iggy_> phantomcircuit: did you bench them all? 1 bad osd can cause problems cluster wide
[6:56] <phantomcircuit> iggy, yup
[6:56] <phantomcircuit> all identical
[6:57] <iggy_> tested network bandwidth between everything?
[6:58] <iggy_> I mean, I'd imagine you'd lose a little with replication, etc... but not that much
[6:59] <iggy_> also, to rule out replication, you could set it to 1 if you are still int he testing phase
[7:01] <phantomcircuit> network is all 10 gbps
[7:10] * tnt (~tnt@83.164-67-87.adsl-dyn.isp.belgacom.be) has joined #ceph
[7:23] * kermin (ca037809@ircip1.mibbit.com) has joined #ceph
[7:23] <kermin> Hi community
[7:23] <kermin> can you tell me How to create subdir and use it to direct data ingest to specific pool?
[7:27] * Ryan_Lane (~Adium@c-67-160-217-184.hsd1.ca.comcast.net) has joined #ceph
[7:33] * sjustlaptop (~sam@68-119-138-53.dhcp.ahvl.nc.charter.com) Quit (Ping timeout: 480 seconds)
[7:48] * scalability-junk (~stp@188-193-211-236-dynip.superkabel.de) Quit (Ping timeout: 480 seconds)
[7:52] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[7:52] * loicd (~loic@magenta.dachary.org) has joined #ceph
[7:56] * kermin (ca037809@ircip1.mibbit.com) Quit (Quit: http://www.mibbit.com ajax IRC Client)
[8:05] * scalability-junk (~stp@188-193-202-99-dynip.superkabel.de) has joined #ceph
[8:07] * deepsa_ (~deepsa@ has joined #ceph
[8:08] * deepsa (~deepsa@ Quit (Ping timeout: 480 seconds)
[8:08] * deepsa_ is now known as deepsa
[8:12] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[8:12] * loicd (~loic@magenta.dachary.org) has joined #ceph
[8:13] * loicd (~loic@magenta.dachary.org) Quit ()
[8:21] * illuminatis (~illuminat@0001adba.user.oftc.net) has joined #ceph
[8:25] * Ryan_Lane1 (~Adium@c-67-160-217-184.hsd1.ca.comcast.net) has joined #ceph
[8:25] * Ryan_Lane (~Adium@c-67-160-217-184.hsd1.ca.comcast.net) Quit (Read error: Connection reset by peer)
[8:38] * illuminatis (~illuminat@0001adba.user.oftc.net) Quit (Quit: WeeChat
[8:42] * eternaleye_ (~eternaley@tchaikovsky.exherbo.org) has joined #ceph
[8:43] * Qu310 (Q@qten.qnet.net.au) has joined #ceph
[8:44] * alexxy[home] (~alexxy@2001:470:1f14:106::2) has joined #ceph
[8:44] * Qten (~qgrasso@qten.qnet.net.au) Quit (Read error: Connection reset by peer)
[8:44] * eternaleye (~eternaley@tchaikovsky.exherbo.org) Quit (Remote host closed the connection)
[8:44] * alexxy (~alexxy@2001:470:1f14:106::2) Quit (Read error: Connection reset by peer)
[8:50] * tnt (~tnt@83.164-67-87.adsl-dyn.isp.belgacom.be) Quit (Ping timeout: 480 seconds)
[9:00] * adjohn (~adjohn@108-225-130-229.lightspeed.sntcca.sbcglobal.net) has joined #ceph
[9:05] * verwilst (~verwilst@dD5769628.access.telenet.be) has joined #ceph
[9:06] * nosebleedkt (~kostas@kotama.dataways.gr) has joined #ceph
[9:09] * yanzheng (~zhyan@jfdmzpr05-ext.jf.intel.com) Quit (Remote host closed the connection)
[9:09] * adjohn (~adjohn@108-225-130-229.lightspeed.sntcca.sbcglobal.net) Quit (Ping timeout: 480 seconds)
[9:16] * Ryan_Lane1 (~Adium@c-67-160-217-184.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[9:20] * loicd (~loic@207.209-31-46.rdns.acropolistelecom.net) has joined #ceph
[9:22] * loicd1 (~loic@207.209-31-46.rdns.acropolistelecom.net) has joined #ceph
[9:28] * loicd (~loic@207.209-31-46.rdns.acropolistelecom.net) Quit (Ping timeout: 480 seconds)
[9:48] * tnt (~tnt@212-166-48-236.win.be) has joined #ceph
[9:53] * roald (~Roald@ has joined #ceph
[10:07] * jbd_ (~jbd_@34322hpv162162.ikoula.com) has joined #ceph
[10:20] <todin> morning #ceph
[10:24] * mtk (~mtk@ool-44c35983.dyn.optonline.net) Quit (Ping timeout: 480 seconds)
[10:34] * timescape (~root@host66-82-static.82-213-b.business.telecomitalia.it) has joined #ceph
[10:40] <nosebleedkt> good morning all
[10:40] <timescape> hi ...
[10:41] <nosebleedkt> I have enabled cephx by setting 'auth supported = ceph' in ceph.conf
[10:41] <nosebleedkt> i also have a client.admin entry in 'ceph auth list'
[10:41] <timescape> I connect here to look for some help...
[10:41] <nosebleedkt> now whatever command i type, i get 'unable to authenticate as client.admin
[10:41] <nosebleedkt> '
[10:42] <nosebleedkt> 'ceph_tool_common_init failed'
[10:42] <nosebleedkt> tnt, joao
[10:43] * timescape (~root@host66-82-static.82-213-b.business.telecomitalia.it) Quit (Quit: Leaving)
[10:48] * morse (~morse@supercomputing.univpm.it) has joined #ceph
[11:14] * MikeMcClurg (~mike@cpc10-cmbg15-2-0-cust205.5-4.cable.virginmedia.com) Quit (Quit: Leaving.)
[11:19] * jtangwk (~Adium@2001:770:10:500:4953:4af3:3fbc:9d8a) Quit (Quit: Leaving.)
[11:20] * jtangwk (~Adium@2001:770:10:500:24d9:d548:19a2:6ea1) has joined #ceph
[11:28] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[11:28] * silversu_ (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Read error: Connection reset by peer)
[11:41] <nosebleedkt> tnt, joao : I have created the user client.admin with a key. Working at the cluster I can issue commands. However I cannot issue commands from client side.
[11:41] <nosebleedkt> any idea?
[11:51] * fc (~fc@home.ploup.net) has joined #ceph
[11:51] * Leseb (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) has joined #ceph
[12:01] * yoshi (~yoshi@p4105-ipngn4301marunouchi.tokyo.ocn.ne.jp) Quit (Remote host closed the connection)
[12:01] * scalability-junk (~stp@188-193-202-99-dynip.superkabel.de) Quit (Remote host closed the connection)
[12:04] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[12:04] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[12:12] * scalability-junk (~stp@188-193-202-99-dynip.superkabel.de) has joined #ceph
[12:13] * Leseb (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) Quit (Quit: Leseb)
[12:24] * glowell (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[12:33] * mooperd (~andrew@dslb-178-000-219-026.pools.arcor-ip.net) has joined #ceph
[12:33] <mooperd> Hey
[12:34] <mooperd> Is there just debian/ubuntu packages?
[12:34] <phantomcircuit> how important is the performance of the mon node?
[12:36] * Leseb (~Leseb@ has joined #ceph
[12:36] <jtang> there are redhat packages
[12:36] * scalability-junk (~stp@188-193-202-99-dynip.superkabel.de) Quit (Ping timeout: 480 seconds)
[12:37] <jtang> i know there are archlinux aur packages as well
[12:37] <jtang> and fedora
[12:37] <mooperd> jtang: Do you know where?
[12:38] * deepsa (~deepsa@ Quit (Ping timeout: 480 seconds)
[12:38] <mooperd> http://ceph.com/docs/master/install/rpm/
[12:38] <mooperd> aa
[12:40] * scalability-junk (~stp@188-193-211-236-dynip.superkabel.de) has joined #ceph
[12:40] * loicd1 (~loic@207.209-31-46.rdns.acropolistelecom.net) Quit (Ping timeout: 480 seconds)
[12:40] * deepsa (~deepsa@ has joined #ceph
[12:48] <jtang> mooperd: see here http://ceph.com/docs/master/install/rpm/
[12:48] <jtang> oh you got it
[12:48] <jtang> the rpm repo isnt the stable version, its the development version thats there as far as i know
[12:50] <mooperd> meeeh
[12:50] <mooperd> ceph.conf
[12:51] <mooperd> Im too hungover for this
[12:51] <mooperd> Im going to configure asterisk insted
[12:52] <jtang> heh
[12:53] <mooperd> so.. I need to set up an mds?
[12:53] <tontsa> configuring asterisk you need to get drunk again
[12:53] <mooperd> tontsa: I usually use methamphetamine
[12:54] * glowell (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) has joined #ceph
[12:54] <mooperd> aaah. So in the config file you define all your oss's in the whole cluster
[12:54] <mooperd> I mean osd
[12:54] <mooperd> s
[12:54] * mooperd looks at his failed lustre installtion with a heavy heart
[12:56] <roald> mooperd, and you´re positive the meth had nothing to do with that?
[12:57] <mooperd> of course. I am a professional
[12:57] <mooperd> Does anyone have a minumum default config for ceph?
[12:57] <roald> touch ceph.conf
[12:58] <mooperd> roald: and I just have to define my osd's
[12:59] <mooperd> how about cluster addresses and public addresses?
[12:59] <roald> mooperd: http://ceph.com/docs/master/start/quick-start/#add-a-configuration-file
[12:59] <roald> this might get you started
[13:00] <joao> I'm pretty sure your osds must know where to contact the monitors
[13:00] <joao> so I'd say that the minimum configuration file must have that info
[13:01] <mooperd> ceph has a learning curve like running into a cliff
[13:01] <mooperd> I like to get the dirt under my fingernails
[13:02] <mooperd> I like it
[13:02] <joao> mooperd, the docs are coming up nicely, and they do have almost every bit of info to smooth that curve
[13:02] <mooperd> half and ounce of heroin and Ill be fine :)
[13:02] <tnt> The 5 minute start guide can get your running in no time.
[13:02] <mooperd> I just have get my head around the concept
[13:03] <mooperd> looks like half gluster half lustre
[13:03] <darkfaded> mooperd: did you read the original ceph paper? it helps
[13:03] <mooperd> Actually not
[13:04] <mooperd> reading now
[13:04] <darkfaded> ok great i found i'm too stupid to use google atm
[13:04] <mooperd> hmm 221 pages.
[13:05] <mooperd> Im probably not going to get through that
[13:06] <joao> that must be sage's thesis
[13:06] <joao> the papers are as long as papers usually go
[13:06] <joao> 16-20 pages maybe
[13:07] <mooperd> http://ceph.com/papers/weil-rados-pdsw07.pdf
[13:07] <mooperd> aaah
[13:07] <mooperd> thats more like it
[13:09] <mooperd> so. Why ceph and not gluster?
[13:09] <mooperd> and why gluster and not ceph?
[13:09] <nosebleedkt> joao, hi
[13:10] <joao> hey there
[13:10] <nosebleedkt> joao, do you know why i get 'unable to authenticate as client.admin' from my client side?
[13:10] <nosebleedkt> I have enabled cephx
[13:10] <nosebleedkt> and i made the keyrings
[13:10] <nosebleedkt> and from cluster side i can issue commands normally
[13:10] <nosebleedkt> but not from client side :(
[13:10] <joao> looking at the monitors logs would be helpful
[13:10] <joao> the leader's logs
[13:11] <nosebleedkt> how to look?
[13:11] <joao> nosebleedkt, first of all, if you don't already have it, I'd suggest cranking up the debug levels on the leader
[13:11] <joao> debug auth = 20
[13:11] <joao> debug ms = 1
[13:11] <nosebleedkt> where do i add this?
[13:11] <joao> that should help you out
[13:12] <joao> the monitor's config file, and then restart the monitor
[13:12] <joao> I'm sure there's a way to do it on the fly, but can't recall it atm
[13:13] <joao> then you should issue the command from the client side, and grep the log for 'client.admin'
[13:13] <joao> if you don't manage to figure it out on your own, just make the logs available to me somewhere and I'll take a look ;)
[13:13] <nosebleedkt> i dont have a config file for monitor
[13:14] <nosebleedkt> i only have ceph.conf
[13:14] <joao> all the same :)
[13:14] <joao> get
[13:14] <joao> get those options under [mon] and you should be okay
[13:14] <nosebleedkt> ah found it
[13:15] <joao> also, what does 'ceph -s' report?
[13:15] <joao> better yet, 'ceph quorum_status'
[13:15] <nosebleedkt> root@cluster:~# ceph -s
[13:15] <nosebleedkt> health HEALTH_OK
[13:15] <nosebleedkt> monmap e1: 1 mons at {a=}, election epoch 0, quorum 0 a
[13:15] <nosebleedkt> osdmap e176: 3 osds: 3 up, 3 in
[13:16] <joao> oh
[13:16] <joao> okay
[13:16] <nosebleedkt> just restarted with logging
[13:16] <joao> just one monitor then
[13:16] <joao> was concerned about who would be the leader so to point you in its log direction
[13:16] <nosebleedkt> ok now where are the log file?
[13:16] <joao> but with only one monitor there's not much to choose from ;)
[13:17] <nosebleedkt> ok i do tail -f /var/log/ceph/ceph-mon.a.log
[13:17] <nosebleedkt> 2012-11-30 14:17:32.147155 b3383b70 10 cephx keyserver: _check_rotating_secrets
[13:17] <nosebleedkt> 2012-11-30 14:17:33.266580 b2980b70 1 -- >> :/0 pipe(0x9515860 sd=21 pgs=0 cs=0 l=0).accept sd=21
[13:17] <nosebleedkt> 2012-11-30 14:17:33.269110 b3b84b70 1 -- <== client.? 1 ==== auth(proto 0 30 bytes epoch 0) v1 ==== 60+0+0 (900162395 0 0) 0x97f8000 con 0x97a8780
[13:17] <nosebleedkt> 2012-11-30 14:17:33.269210 b3b84b70 1 -- --> -- auth_reply(proto 0 -95 Operation not supported) v1 -- ?+0 0x9605540 con 0x97a8780
[13:17] <joao> that probably won't be much help; you need to figure out the specific events
[13:18] <joao> you'll want to grep for 'client.admin' after the failed auth message
[13:18] <nosebleedkt> auth_reply : operation not support
[13:18] <nosebleedkt> auth_reply : operation not supported
[13:18] <joao> hmm
[13:18] <joao> do you have cephx enabled?
[13:18] <nosebleedkt> in cluster config yes
[13:18] <nosebleedkt> i must do it in client config too ?
[13:19] <joao> what version are you running?
[13:19] <nosebleedkt> 0.48
[13:19] <nosebleedkt> argonaut
[13:19] <joao> then I think so
[13:19] <joao> cephx only got enabled by default back in 0.54 iirc
[13:19] <nosebleedkt> oh lol
[13:19] <nosebleedkt> it worked 1
[13:19] <nosebleedkt> !
[13:20] <joao> great :)
[13:20] <nosebleedkt> yeah :D
[13:20] <nosebleedkt> fun
[13:20] <nosebleedkt> :)
[13:20] * illuminatis (~illuminat@0001adba.user.oftc.net) has joined #ceph
[13:26] <phantomcircuit> it seems to me that the journal playback is happening too agressively
[13:27] <phantomcircuit> is there someway to tune that?
[13:27] <phantomcircuit> (journal is on an ssd, osd is on a conventional hdd, hdd is maxing out IOPS but not throughput)
[13:27] <nosebleedkt> joao, but also i need to have a copy of the key on the client side too... how is this done automatically? or it can't ?
[13:28] <joao> what do you mean?
[13:28] * ramsay_za (~ramsay_za@ Quit (Quit: Leaving)
[13:30] <nosebleedkt> root@cluster:~# cat /etc/ceph/keyring
[13:30] <nosebleedkt> [client.admin]
[13:30] <nosebleedkt> key = AQCEkLhQaGBYLxAASwDlOHcbd/bPzytNdJt0KA==
[13:31] <nosebleedkt> if i dont have that file in my client computer
[13:31] <nosebleedkt> the client cannot be authnticated
[13:31] <joao> yep
[13:31] <joao> you got to authenticate somehow
[13:31] <nosebleedkt> so I need to copy that keyring file to all the clients
[13:31] <joao> I don't know if there's a better way, but that's how I'd do it
[13:32] <nosebleedkt> this procedure should be done automatically or manually by the cluster admin /
[13:32] <nosebleedkt> ?
[13:32] <joao> if I felt the need to have multiple clients, that is
[13:33] <nosebleedkt> let's assume there is a guy with a computer and he is using my cluster.
[13:33] <nosebleedkt> His computer should have that keyring file
[13:33] * tnt (~tnt@212-166-48-236.win.be) Quit (Ping timeout: 480 seconds)
[13:33] <joao> yeah
[13:33] <joao> not sure if you can issue multiple users/keys though
[13:34] <nosebleedkt> I have to give to every guy that uses my cluster
[13:34] <nosebleedkt> that file?
[13:34] <joao> never thought about it tbh
[13:34] <joao> yeah
[13:34] <nosebleedkt> ermm
[13:34] <joao> think so
[13:34] <nosebleedkt> how is a cluster supposed to be used by big companies/
[13:34] <nosebleedkt> ?
[13:36] <joao> you should only need that key for administration tasks, afaik
[13:38] <joao> I might not be the best person to talk to about how big deployments work though
[13:39] <joao> and by the sudden hunger, looks like it must be lunch time
[13:39] <joao> bbiab
[13:39] <phantomcircuit> hmm
[13:41] <nosebleedkt> joao, have a good launch :)
[13:41] <phantomcircuit> sdc 158.00 12.00 7821.50 12 7821
[13:41] <phantomcircuit> Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
[13:41] <phantomcircuit> hmm
[13:41] <phantomcircuit> definitely more iops than my poor conventional hdd can handle :(
[13:43] <jtang> hmm ya know what some better docs on the authentication in ceph would be nice
[13:43] <jtang> some "workflow" examples would be nice
[13:44] <jtang> im still a little confused as to where to execute commands and where to put the generated keys
[13:45] * tnt (~tnt@83.164-67-87.adsl-dyn.isp.belgacom.be) has joined #ceph
[13:45] <phantomcircuit> jtang, personally i just copied the keyring to every client
[13:45] <phantomcircuit> of course that isn't the designed way of doing it but well...
[13:45] <jtang> yea i do like the idea of controlling what clients can and cant do
[13:45] <jtang> :)
[13:45] <jtang> just thinking about features that would be nice to have
[13:45] <jtang> are quotas on the roadmap?
[13:46] <jtang> seems like the disk used reporting feature in cephfs looks like you can have a hierachy of quotas
[13:46] <jtang> that would be quite cool
[13:46] <jtang> it would map very well the way we run our compute clusters in terms of allocating cpu time
[13:49] <jtang> actually that would be cool to be able to mount a directory in cephfs, have a quota which reports back as being the diskavailable to the client
[13:50] <jtang> that way when i give a mount to a customer they only see what they paid for and not the entire free space in my system
[13:50] <dweazle> i would welcome a means to limit iops for rbd clients
[13:50] <phantomcircuit> hmm i just realized im running 0.49 which is gentoo stable
[13:50] * morse (~morse@supercomputing.univpm.it) Quit (Remote host closed the connection)
[13:50] <phantomcircuit> just how ridiculously old is this...
[13:51] <darkfaded> i still need to pick an OS for the class next week
[13:51] <jtang> heh
[13:51] <jtang> there is a class?
[13:51] <jtang> another ceph day?
[13:51] <darkfaded> either debian7 or fedora16 (which i already have ready) or nightmare-buntu
[13:51] <phantomcircuit> gentoo! if you start compiling now it'll be ready!
[13:51] * phantomcircuit runs
[13:52] <darkfaded> jtang: no i'm doing storage classes and have added ceph to the schedule back in feb
[13:52] * calebmiles (~caleb@65-183-137-95-dhcp.burlingtontelecom.net) has joined #ceph
[13:52] <jtang> ah i see
[13:52] <jtang> nothing really wrong with ubuntu
[13:52] <darkfaded> i took the materials over and it was all outdated, so i refreshed things
[13:52] <jtang> tbh the debian class of OS's is okay
[13:52] <darkfaded> now with LIO and ceph instead of old-stuff
[13:52] <jtang> if you dont have enterprise hardware that needs specific kernels
[13:52] <dweazle> phantomcircuit: nah, just spawn 100 amazon instances and use distcc and ccache and it'll be done tomorrow
[13:53] <darkfaded> dweazle: hahaha
[13:53] <jtang> or if you dont mind fiddling with kernel packages which change all the time
[13:53] <phantomcircuit> dweazle, i bet there is someone who has done that
[13:53] <phantomcircuit> i bet there is someone who does that *regularly*
[13:53] <jtang> ccache on a tmpfs is good
[13:53] <darkfaded> jtang: flashcache, lvm, thin provisioning lio and ceph in one distro without custom compiles is almost no-way
[13:53] <dweazle> hehe yeah, especially those running kde!
[13:54] <darkfaded> ccache is always good :>
[13:54] * morse (~morse@supercomputing.univpm.it) has joined #ceph
[13:54] <phantomcircuit> i started building kde once and just gave up after 3 days :|
[13:54] <jtang> darkfaded: ah ok
[13:54] <jtang> i use dwm on linux
[13:54] <jtang> :P
[13:54] <jtang> its probably not for everyone though
[13:54] <dweazle> i use mingetty on linux
[13:54] <jtang> i miss my linux workstation, (using a mac these days)
[13:55] <jtang> i just sat through evaluating GPFS and iRODS formally for the project im working on
[13:55] <jtang> hadoop/hdfs is being done rightnow
[13:55] <jtang> and ceph is next on the hit list
[13:55] <jtang> shame i cant just make an arbitrary decision
[13:55] <jtang> :)
[13:56] <phantomcircuit> darkfaded, flashcache is very nice, i just tested it as writeback cache and realized it's actually slower if it's on the same ssd volume as the journal
[13:56] <phantomcircuit> sequential throughput of the ssd is effectively split between the journal and the disk
[13:56] <darkfaded> they're getting contented?
[13:56] <phantomcircuit> yeah
[13:57] <darkfaded> so did you use two ssds and see if it flies now?
[13:57] <jtang> phantomcircuit: you mean you tried using flashcache for an OSD?
[13:57] <darkfaded> jtang: or maybe ontop of /dev/rbd
[13:57] <jtang> ah okay
[13:57] <phantomcircuit> no for an osd
[13:57] <phantomcircuit> just to see if it would do anything
[13:57] <jtang> two slightly different cases
[13:57] <phantomcircuit> the answer is no
[13:57] <darkfaded> so the osd is on a disk and the journal + flashcache on an ssd each?
[13:57] <jtang> we're thinking of getting some fusion-io cards here to test with
[13:58] <jtang> got to hit some vendors to loan us a few to play with
[13:58] <darkfaded> phantomcircuit: did you check the flashcache stats? did it even bother caching?
[13:58] <phantomcircuit> yeah it was actually doing quite a bit of caching
[13:58] <phantomcircuit> but wasn't anywhere near being worth it
[13:59] <darkfaded> hehe
[13:59] <jtang> the io probably was too random?
[13:59] <phantomcircuit> the box has 2 ssds and two hdds
[13:59] <darkfaded> i only used write-around mode and that also doesnt always perform well
[13:59] <phantomcircuit> i have another identical system which doesn't use ceph and just has flashcache for qcow2 images
[13:59] <phantomcircuit> running that in writeback mode it's actually incredibly fast
[14:00] <phantomcircuit> but it wasn't designed for that so you have to stop everything and security erase the ssds on a schedule or performance will consistently degrade
[14:00] <phantomcircuit> flashcache doesn't do trim :|
[14:01] <jtang> i discovered you can write a crush rule to read from certain pools first
[14:01] <darkfaded> phantomcircuit: they run it on fusion-io, it does not really do anything smart for less reliable / slower ssds
[14:01] <dweazle> is trim support really that important these days with newer gen ssd's?
[14:01] <darkfaded> (unfortunately)
[14:01] <jtang> i was thinking about looking into some forms of tiering in ceph
[14:01] <phantomcircuit> dweazle, when you're using almost 100% of the device it's *very* important
[14:02] <phantomcircuit> darkfaded, well when you have 50+ active vps' constantly flushing disk cache it helps just to be able to absorb the random writes into sequential writes
[14:02] <darkfaded> phantomcircuit: i bet
[14:02] <phantomcircuit> the sequential throughput of the ssd i have as writeback is actually about half that of the hdd
[14:02] <phantomcircuit> but it's still worth it
[14:02] <phantomcircuit> (ie it's the only reason anything works at all )
[14:02] <darkfaded> LOL debian just wants to install texlife etc for targetcli. iscsi over latex, i presume
[14:03] <darkfaded> phantomcircuit: hmmm, offtopic, i have made a flashcache check for nagios some time, would you mind helping me with some real life stats?
[14:04] <darkfaded> and besides, i should get back to ceph anyway
[14:05] <phantomcircuit> darkfaded,
[14:05] <phantomcircuit> http://pastebin.com/raw.php?i=NEDYYpWP
[14:05] <nhm> good morning #ceph
[14:07] <darkfaded> thanks
[14:07] <jtang> nhm: where are you based?
[14:07] <nhm> jtang: Minneapolis, MN
[14:07] <jtang> it seems awefully early for someone in SF?
[14:07] <jtang> ah i see
[14:08] <phantomcircuit> jtang, it's only 5am
[14:08] <jtang> so you're only about 6hrs behind GMT0
[14:08] <phantomcircuit> >.>
[14:08] <phantomcircuit> <.<
[14:08] <jtang> heh
[14:08] <nhm> jtang: yeah, and most of those guys don't get in until like 9am PST anyway. ;)
[14:08] <nhm> jtang: working from home has its advantages. :)
[14:08] * yanzheng (~zhyan@ has joined #ceph
[14:09] <jtang> oh
[14:10] <jtang> hmmm
[14:10] <jtang> i must look at S3 at somepoint
[14:10] <jtang> maybe i shuold just get our project to use the s3 interface
[14:11] <nhm> jtang: how about librados? ;)
[14:11] <jtang> i was talkin to rweeks about it
[14:11] <jtang> librados looks pretty cool
[14:11] <jtang> it looks like i can implement microservices with it
[14:11] <jtang> unfortunately im not the person doing the storage work :P
[14:11] <jtang> im only telling others to look at it
[14:12] <jtang> so i guess my co-workers needs to select things they are comfortable with (posix and s3 might be the way to go)
[14:12] <jtang> i wonder if anyone has thought about workflows sitting ontop of librados
[14:13] <jtang> that'd be pretty cool to have a bunch of microservices sitting on top of librados, then have workflows in the form of a perl/ruby/whatever script to do clever things
[14:14] <nhm> jtang: I haven't, but I've been burried in our backend code looking for low-hanging performance fruit.
[14:14] <nhm> well, not really in the code unfortunately.
[14:15] <jtang> heh
[14:15] <jtang> nhm: you should push to fix up the issues with adding in new OSD's
[14:15] <jtang> to stop new OSD's being flooded with replicated data and new data
[14:15] <jtang> i think that would be a nice low hanging thing to fix
[14:16] <nosebleedkt> [21243.090207] libceph: client0 fsid 275e3f6e-96b2-4894-84db-c56e7f9db7b5
[14:16] <nosebleedkt> [21243.090594] libceph: no secret set (for auth_x protocol)
[14:16] <nosebleedkt> [21243.091168] libceph: error -22 on auth protocol 2 init
[14:16] <jtang> right now i think the recommendation is to manually change the weights
[14:16] <nosebleedkt> trying to: rbd map image --pool rbd --name client.admin
[14:16] <nhm> jtang: ah, some kind of throttling so it doesn't get so hammered?
[14:16] <jtang> nhm: yea
[14:17] <nhm> jtang: how bad it is in practice?
[14:17] <jtang> i would have thought doing something simple like when an osd is put in, if its not more than 50% full rate limit it
[14:17] <jtang> or something like that
[14:17] <jtang> we have two pods
[14:17] <jtang> and if one fails we need to shift up 60tbs of data
[14:18] <jtang> thats sort of not nice if the system is in production and replication is happening with live IO going to the pods
[14:18] <jtang> we do have the OSDs running in the back end on a private network
[14:18] <jtang> but the disks/controllers themselves kinda suck
[14:19] <jtang> so instead of getting some thrashing getting a smooth ramping up of an OSD would be preferable
[14:19] <jtang> that way at least performance is abit more predictable over all
[14:19] <jtang> spikes are bad :P
[14:19] <nhm> jtang: Yeah, definitely. I think we tend to think of ceph clusters as having more/smaller osd servers, in the case where you've got few big servers, I imagine replication pain could be intense.
[14:20] <jtang> i guess the use case/problem might exist in big clusters as well
[14:20] <jtang> how does dreamhost deal with down'd nodes?
[14:21] <jtang> does a fresh osd just get blasted by 100 other machines to fill the osd?
[14:21] <nhm> jtang: Good question. I should find out what policies they've set for replication.
[14:21] <jtang> or pick an arbitrary number
[14:21] <jtang> i guess stuff like that should be in a best practice document ;)
[14:21] <nhm> jtang: It's been a while since I talked to them about their deployment.
[14:22] * mtk (~mtk@ool-44c35983.dyn.optonline.net) has joined #ceph
[14:22] <jtang> it's certainly different to what GPFS does anyway in terms of replication
[14:22] * jtang has to kick gpfs to replicate data
[14:24] <jtang> stuff like this used to keep me up at night
[14:24] * calebmiles1 (~caleb@65-183-137-95-dhcp.burlingtontelecom.net) has joined #ceph
[14:24] <jtang> btw - http://digitalcollections.tcd.ie/ - checkout the book of kells!
[14:25] <nhm> jtang: Thinking about this more, I wonder if it really is a policy decision. IE, is it more important to you to have a slow cluster that prioritizes healing, or a faster cluster that isn't healing as fast (and potentially could have a higher likely hood of losing data if another node failed during the healing process).
[14:25] * calebmiles1 (~caleb@65-183-137-95-dhcp.burlingtontelecom.net) Quit ()
[14:25] * loicd (~loic@magenta.dachary.org) has joined #ceph
[14:25] <jtang> nhm: i thoght that in a big enough cluster its less about protecting data if you have enough replicas, but more about balancing the data so you can suffer a larger failure that otherwise possible
[14:26] <nhm> jtang: it's too early for me. You are right, the cluster is already healthy, you are talking about re-distributing back to the OSD that was out.
[14:26] <jtang> given that ceph is replicating data automatically (if the system is designed correctly) then does it matter if an osd gets slowly brought back to improve overall performance?
[14:27] <nhm> jtang: this is why I shouldn't have technical discussions right after waking up in the morning.
[14:27] <jtang> heh, well get a coffee ;)
[14:28] <jtang> yea maybe a bw limiter for the OSD operations might do the trick
[14:28] <jtang> so you dont flood the osd's
[14:29] <jtang> i could probably do that with iptables
[14:29] <jtang> hrmmm
[14:29] * jtang ponders
[14:30] * calebmiles (~caleb@65-183-137-95-dhcp.burlingtontelecom.net) Quit (Ping timeout: 480 seconds)
[14:30] <nhm> jtang: yeah, I think the important point is to have ceph differentiate replication traffic due to healing and replication traffic due to rebalancing. I honestly don't know the extent to which we do that.
[14:32] <jtang> they're nice to have features, probably not too critical for the usage of ceph
[14:32] <jtang> if i was in your shoes, it's probably down there in the long term roadmap
[14:32] <jtang> :)
[14:33] <jtang> as devops person i'd be asking those questions
[14:33] * deepsa (~deepsa@ Quit (Quit: ["Textual IRC Client: www.textualapp.com"])
[14:33] <jtang> as consumer of the system that provides storage from a ceph install i wouldnt care less
[14:34] <darkfaded> jtang: amazon going down last year in a "rebalancing storm" shows that devops persons do not ask that question ;)_
[14:36] <jtang> heh
[14:36] <jtang> does that make me a non-standard devops person?
[14:36] * jtang likes to ask questions
[14:36] <jtang> i often dont get answers though
[14:37] * jtang has also been burnt but a few storage systems before too
[14:37] * jtang eyes lustre
[14:38] <jtang> thank god they got rid of them xml configuration files
[14:38] <nhm> jtang: I used to be in charge of some lustre storage too.
[14:39] <jtang> we had one cluster with lustre
[14:39] <jtang> and it wasnt pleasant to admin
[14:39] <jtang> performance was good though
[14:39] <nhm> jtang: you just have to learn to like it. ;)
[14:39] <jtang> i spent a few months working with some guy on the debian packaging years ago
[14:39] <jtang> i learnt to dislike it, the build system was horrid
[14:40] <jtang> the configuration was horrid
[14:40] <jtang> the setup was horrid
[14:40] <jtang> i learnt to not like it ;)
[14:40] <nhm> jtang: How much did you have deployed?
[14:40] <jtang> about 60tb of disk about maybe 4yr ago
[14:41] <jtang> just when it was v1.6 going to 1.8GA
[14:41] <nhm> yeah, 1.6 was pretty awful, and there were some bugs in early 1.8 releases.
[14:41] <jtang> we ran on the first DDN6600 series of machines (which are now pretty much the SFA machines)
[14:41] <jtang> that was funny
[14:41] <nhm> Ah, ok. We had a 6620, but we didn't use it for lustre.
[14:42] <jtang> we had a pre-production 6600 series
[14:42] <jtang> yea thats the one the 6620
[14:42] <nhm> Our lustre storage was Scalable Informatics, and then Terascala on Dell hardware.
[14:42] <jtang> we got ours from clustervision
[14:42] <jtang> and the lustre install was erm... homebrew
[14:43] <jtang> i hated the rolly thing at the back
[14:43] <jtang> the cables kept on getting cut accidently
[14:43] * agh (~Adium@gw-to-666.outscale.net) has joined #ceph
[14:43] <jtang> or pulled out from the controllers
[14:43] <jtang> the firmware was buggy, it locked up allthe time
[14:43] <jtang> we ended up getting it replaced with a ddn9900
[14:44] <nhm> I never did much with our 6620. They were using it for some genomics project that I wasn't involved in.
[14:45] <nhm> We had about 800-900TB of lustre storage.
[14:45] <jtang> yea thats sorta what we got it for as well
[14:45] <nhm> in total.
[14:45] <jtang> we had it filled with SAS
[14:45] <nhm> We also had like 700TB of CXFS
[14:45] <jtang> cool
[14:45] <jtang> we're too small here to have *that* storage
[14:45] <nhm> CXFS is kind of scary too. ;)
[14:46] <jtang> we had CXFS on an altix here :)
[14:46] <jtang> its not too bad
[14:46] <jtang> its better than lustre
[14:46] <nhm> jtang: we had it on an Altix UV1000. I got the feeling our deployment was the first time they tried it. ;)
[14:46] <jtang> UV is over-rated
[14:46] <jtang> and it came too late
[14:46] <nhm> jtang: very
[14:47] <Meyer__> oh, UV1000, they are nice
[14:47] <jtang> if it came along 5yrs earlier, it would have rocked
[14:47] <jtang> but things like scalemp is just a cheaper alternative
[14:47] <nhm> jtang: we got a big grant to buy one. I wish we would have just bought 5-6 big memory nodes from someone else. Very few people were using more than 512GB of ram anyway.
[14:47] <jtang> heh
[14:47] <jtang> right time to go
[14:48] <jtang> i have a half day in work today
[14:48] <jtang> and all my tests have passed!
[14:48] <nhm> Meyer__: It took a long time for ours to get stable enough to pass acceptance.
[14:48] <agh> hello to all
[14:48] <nhm> jtang: good deal. Have a good one. :)
[14:48] <jtang> so time to reward myself by going to my friends wedding and making him run around dancing!
[14:49] <jtang> so his wife can laugh at how much of a bunch of idiots we are...
[14:49] <jtang> later
[14:49] <agh> i'm testing Ceph, and it's seems to be very cool. But I have a question : is it possible to change the size of the journal online ? without disturbing the cluster ?
[14:49] <nhm> Meyer__: Also, Numalink5 wasn't much faster than IB for us, and since it was more or less a 2D torus, it was slower overall than our IB cluster with a fat-tree switch.
[14:49] * aliguori (~anthony@cpe-70-123-145-75.austin.res.rr.com) has joined #ceph
[14:51] <nhm> agh: hrm, there may possibly be a way to do it with btrfs, but I wouldn't attempt it without guidance from someone who actually knows what they are talking about. With XFS and EXT4 I don't think you can.
[14:52] <agh> nhm: thanks. So we need to be very carful at the begining when choosing our journal size, that's it ?
[14:53] <agh> by the way, do you know if there is some benchmarks or article about some real Ceph cluster ? (with minimum 100 nodes ?)
[14:54] <nhm> agh: Yeah. The bigger you make it, the longer you can absorb bursts of random writes beforfe you have to write them out to the underlying filesystem.
[14:54] <Meyer__> nhm: Well, we replaced it with a if i recall correctly Altix 320, so it was quite the improvement. And it was the "multi purpose" HPC system. This particular client had bunch of other HPC systems for other type of calculations
[14:54] <Meyer__> nhm: So, in this use case the UV1000 actually did a good job. But it was not "fully stacked". 384 cores and 2TB of ram or so
[14:55] <nhm> agh: eventually it has to get written out though.
[14:55] <nhm> Meyer__: ah, ok. We had about 1100 cores and 3TB of ram.
[14:56] <nhm> Meyer__: So the NL5 network was really oversubscribed.
[14:56] <Meyer__> nhm: It was mainly purchased to run stuff that needed a "single memory space on one system".
[14:57] <nhm> Meyer__: 384 cores was probably like 16 system board pairs? That's much more manageable.
[14:57] <Meyer__> nhm: and in addition was a generic whatever else they figured out they wanted to do
[14:57] <Meyer__> nhm: Yes, it was based on 6 core cpu's so if i recall correctly it was 16 system boards
[14:58] <nhm> I think you got 4 NL5 connections per board-pair, so 16 of those wouldn't be too bad to connect.
[14:59] <nhm> Meyer__: we had 48 board-pairs in ours with the same 4 links per board-pair.
[15:00] <agh> do you use Ceph in prod ?
[15:00] <nhm> The topology was kind of insane.
[15:00] <Meyer__> nhm: Oh, I would assume that was a bit more oversubcribed
[15:01] <nhm> agh: I work for Inktank, so er, sort of? :) Dream Host is our sister company though and has something like 7PB in production now I think.
[15:01] * nhorman (~nhorman@hmsreliant.think-freely.org) has joined #ceph
[15:01] <Meyer__> thats quite a lot :)
[15:01] <agh> with Btrfs ?
[15:01] <agh> I'm afraid of Btrfs
[15:01] <nhm> agh: XFS
[15:01] <agh> Ah ok
[15:02] <agh> nhm: I'm ZFS fanboy… so Btrfs… hum :)
[15:02] <nhm> agh: Way down somewhere on my list is to do some ZFS testing.
[15:02] <nhm> agh: It's just hard to prioritize it if it won't ever be in the kernel. :/
[15:03] <agh> nhm: interesting. under linux with ZOL or under FreeBSD ?
[15:03] <nhm> I was going to try ZOL. Not sure where the LLNL guys are at these days, but for a while I was trying to follow it.
[15:03] <agh> nhm: I understand. I will never understand the goal of btrfs : doing like ZFS, but.. worse
[15:04] <agh> nhm: ZOL is works fine. But unfortunatly, the perfs are not (yet) there
[15:04] <nhm> agh: between XFS, EXT4, and BTRFS, we are fastest on BTRFS if it's a fresh file system. In the past performance has degraded fairly quickly on aged filesystems though.
[15:05] <agh> nhm: ok. and, BTW, we are going to buy some nodes to make a lab for Ceph
[15:05] <nhm> agh: Excellent. :)
[15:05] <nhm> agh: What kind of usecase?
[15:05] <agh> nhm: we will go to SuperMicro. Do you have an advice ? How many nodes to have a realistic lab ?
[15:06] <agh> nhm: cloud
[15:06] <agh> nhm: > 10PB
[15:06] <nhm> agh: Have you read the controller write throughput articles yet?
[15:06] * calebmiles (~caleb@65-183-137-95-dhcp.burlingtontelecom.net) has joined #ceph
[15:06] <agh> nhm: yes
[15:06] <agh> nhm: but not in details, :$ I've to print it to read it in bed :)
[15:07] <nhm> agh: ok, beyond that, be aware that we top out at around 1.4GB/s per node (2.8GB/s internally counting the journal) write now with niave tuning.
[15:07] <agh> so how many nodes, how many disks per osd-hosts, SAS or sata ? etc etc
[15:08] <nhm> agh: It might be that with tuning we could push things higher. DreamHost is using 12-drive 2U systems right now but is exploring other options.
[15:09] <agh> nhm: so your advice is to have more osd hosts with few disks, whereas having less hosts with more disks (sorry for my english, i'm not fluent ;))
[15:09] <nhm> agh: SAS vs SATA is probably less of an important question for ceph and more of an important question depending on your controller and expanders. enterprise vs non-enterprise is always a contentious debate.
[15:10] <agh> i've a lot of questions, sorry, but, do you have a stress-test or a benchmark procedure to test Ceph ?
[15:10] <nhm> agh: ~12-drive nodes are probably the best tested right now. We've started playing on denser systems, but you should probably only buy them for density and not expect to be pushing 2+GB/s out of them yet.
[15:11] <agh> nhm: ok. it was also my thought : 1 disk / core so 12 cores => 12 disks. quite good isn't it ?
[15:11] * mdawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) has joined #ceph
[15:11] <nhm> agh: We've got a couple of tools we've written for lower level ceph benchmarking. We tend to use fio for disk and RBD testing.
[15:11] <Meyer__> Why is throughput so important? I would assume most use cases has more need for IOPS?
[15:12] <agh> nhm: fio. OK. i note.
[15:12] <nhm> agh: I think we are still recommending 1GHZ per OSD, so you might be able to get away with 6 fast cores for a 12-drive system.
[15:13] <agh> nhm: ok great
[15:13] <nhm> Meyer__: Depends on the customer. We get requests for both.
[15:14] <agh> nhm: last but not least, is it possible to have 2 journals disks : 2 SSD. If one fails, the other one take the relay
[15:14] <nhm> Meyer__: The HPC crowd really wants sequential throughput. The guys that want to throw databases on RBD want IOPs. Makes our lives hard. ;)
[15:15] <nhm> agh: journals are just partitions on a block device or files on a filesystem. You can raid-1 them behind the scenes if you want.
[15:16] <nhm> agh: typically people don't do that though, and just try to use replication over lots of nodes in different racks since that's what Ceph is good at.
[15:17] <agh> nhm: ok. thank you very much for you answer
[15:17] <nosebleedkt> tnt, are you here?
[15:17] <nhm> ok, gotta run for a bit. ttyl guys
[15:18] <Meyer__> nhm: Yes, thats true of course. I did not consider the HPC crowd :)
[15:19] <Meyer__> nhm: Do you have any numbers on how many IOPS such a 12 drive 2U system can deliver with for exampel 8-16k random reads?
[15:24] * maxiz_ (~pfliu@ Quit (Ping timeout: 480 seconds)
[15:24] <nosebleedkt> does anyone know something about this error?
[15:24] <nosebleedkt> [25227.827966] libceph: no secret set (for auth_x protocol)
[15:24] <nosebleedkt> [25227.828952] libceph: error -22 on auth protocol 2 init
[15:24] <nosebleedkt> I get it when i enable cephx
[15:25] <nosebleedkt> while trying to do this: rbd map image --pool rbd --name client.admin
[15:26] * The_Bishop__ (~bishop@e179017250.adsl.alicedsl.de) has joined #ceph
[15:33] * The_Bishop_ (~bishop@f052099189.adsl.alicedsl.de) Quit (Ping timeout: 480 seconds)
[15:52] * nosebleedkt (~kostas@kotama.dataways.gr) Quit (Quit: Leaving)
[15:54] * MikeMcClurg (~mike@firewall.ctxuk.citrix.com) has joined #ceph
[15:54] * scuttlemonkey (~scuttlemo@c-69-244-181-5.hsd1.mi.comcast.net) Quit (Read error: Connection reset by peer)
[15:58] * scuttlemonkey (~scuttlemo@c-69-244-181-5.hsd1.mi.comcast.net) has joined #ceph
[15:58] * ChanServ sets mode +o scuttlemonkey
[16:06] * ScOut3R (~ScOut3R@ has joined #ceph
[16:07] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[16:10] * ScOut3R (~ScOut3R@ Quit (Remote host closed the connection)
[16:14] * noob2 (~noob2@ext.cscinfo.com) has joined #ceph
[16:21] * yanzheng (~zhyan@ Quit (Remote host closed the connection)
[16:33] * illuminatis (~illuminat@0001adba.user.oftc.net) Quit (Quit: WeeChat
[16:41] * Dr_O (~owen@00012c05.user.oftc.net) Quit (Quit: Ex-Chat)
[16:48] * yanzheng (~zhyan@ has joined #ceph
[16:56] * verwilst (~verwilst@dD5769628.access.telenet.be) Quit (Quit: Ex-Chat)
[17:02] * yanzheng (~zhyan@ Quit (Ping timeout: 480 seconds)
[17:02] * loicd (~loic@jem75-2-82-233-234-24.fbx.proxad.net) has joined #ceph
[17:05] * ircolle (~ian@c-67-172-132-164.hsd1.co.comcast.net) has joined #ceph
[17:14] * jlogan1 (~Thunderbi@2600:c00:3010:1:e12a:776f:2a6d:8a8) has joined #ceph
[17:29] * senner (~Wildcard@68-113-232-90.dhcp.stpt.wi.charter.com) has joined #ceph
[17:54] * mdawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) Quit (Read error: Connection reset by peer)
[17:54] * mdawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) has joined #ceph
[18:09] * rweeks (~rweeks@c-98-234-186-68.hsd1.ca.comcast.net) has joined #ceph
[18:12] * andreask (~andreas@212095007076.public.telering.at) has joined #ceph
[18:17] <joao> sagewk, around?
[18:20] <darkfaded> http://ceph.com/docs/master/rados/operations/add-or-rm-osds/ puts the osd in /var/lib ... and no dir setting in ceph.conf, is that a best practice default now and will just work, or is it hardcoded, or is it just missing in the doc?
[18:24] * stxShadow (~Jens@ip-178-203-169-190.unitymediagroup.de) has joined #ceph
[18:25] <joao> darkfaded, hardcoded default
[18:25] <darkfaded> joao: thanks :)
[18:26] <joao> as a means of standardizing installs, but you can define whatever you want on the conf
[18:27] <darkfaded> i like standardized paths
[18:27] <darkfaded> i hate data in /var
[18:27] * benpol (~benp@garage.reed.edu) has joined #ceph
[18:27] <darkfaded> but i wont cry if it gets me a standardized path :)
[18:30] <darkfaded> so if i specify [osd.9] it'll be looking for /var/lib/ceph/ceph-9 based on the number?
[18:30] * mtk (~mtk@ool-44c35983.dyn.optonline.net) Quit (Remote host closed the connection)
[18:31] <darkfaded> (just need to make sure i know how my magic spells work)
[18:31] * match (~mrichar1@pcw3047.see.ed.ac.uk) Quit (Quit: Leaving.)
[18:33] <darkfaded> make that /var/lib/ceph/osd/ceph-9
[18:35] <joao> yep
[18:35] <benpol> There used to be a nice "Cluster Operations" section in the ceph docs at http://ceph.com/docs/master/ I know most of the docs are still there (individual pages that I bookmarked are still accessible). Has that section been deprecated or replaced?
[18:36] <joao> benpol, still there: http://ceph.com/docs/master/rados/operations/
[18:37] <benpol> joao: thanks, good to know. the section used to be at the top level of the table of contents removing it makes it harder to find.
[18:38] <rweeks> it's just under the RADOS section now
[18:38] <darkfaded> and if i have mon initial members = acht, drei, fuenf
[18:39] <darkfaded> that does not replace a mon.a , mon.b , mon.c section, right?
[18:39] * Leseb (~Leseb@ Quit (Quit: Leseb)
[18:40] <benpol> rweeks: got it, thanks (IMHO it should be more readily discoverable)
[18:41] * rweeks pages rturk or scuttlemonkey
[18:42] * andreask (~andreas@212095007076.public.telering.at) has left #ceph
[18:43] <roald> darkfaded, they will become mon.acht, mon.drei, ...
[18:44] <roald> sections are named [$name], and $name = $type.$id ($type being mon, $id being ´acht´ etc)
[18:45] <yehuda_hm> gregaf, sagewk: wip-3529, wip-3535 can use another pair of eyeballs
[18:51] <darkfaded> roald: i think my head is falling off soon anyway. if i make a section named mon.c then it should be looking for /var/lib/ceph/mon/mon-c the way i understand it. i trust it'll be as you say but i donz get why
[18:52] <scuttlemonkey> wha?
[18:53] <scuttlemonkey> the cluster operations thing?
[18:54] <rweeks> yes
[18:54] <scuttlemonkey> yeah, as I understand it the doc refactor is coming
[18:54] <scuttlemonkey> john wanted to get a few other things polished before it was all rolled out and pretty
[18:54] <rweeks> ok
[18:55] <rweeks> just figured you guys should always get poked when there is feedback
[18:55] <scuttlemonkey> there are several 301s in there already, and more coming
[18:55] <scuttlemonkey> yah, is helpful
[18:55] <scuttlemonkey> thanks
[18:55] <scuttlemonkey> I have so many irc windows open it's nice to see it go blue when there is something needing attn
[18:57] * chutzpah (~chutz@ has joined #ceph
[19:00] * jbd_ (~jbd_@34322hpv162162.ikoula.com) has left #ceph
[19:02] <gregaf> yehuda_hm: it's on my list
[19:09] * miroslav (~miroslav@173-228-38-131.dsl.dynamic.sonic.net) Quit (Quit: Leaving.)
[19:13] <benpol> Thanks joao, rweeks, scuttlemonkey, those docs are the primary resource for people getting their feet wet in the ceph world :)
[19:18] <darkfaded> *head desk*
[19:18] <darkfaded> 2012-11-30 19:18:07.517688 7f1d5d78e780 -1 OSD::mkfs: FileStore::mkfs failed with error -16
[19:19] <darkfaded> how can i re-run mkcephfs and please not have it try to mount the osd directories
[19:19] <darkfaded> or format them, if they are directories, after all
[19:20] <darkfaded> http://paste.ie/view/4d6fa994
[19:20] <darkfaded> this is the current config, which has not yet worked (so assume its wrong hehe)
[19:20] * senner (~Wildcard@68-113-232-90.dhcp.stpt.wi.charter.com) Quit (Quit: Leaving.)
[19:21] * eternaleye_ is now known as eternaleye
[19:21] <scuttlemonkey> so what empire am I choosing?
[19:22] <scuttlemonkey> sry, I need to consolidate windows
[19:22] <scuttlemonkey> in other news...we need fewer trello installs :P
[19:26] <gregaf> darkfaded: you can't re-run mkcephfs
[19:26] <gregaf> that's why we are trying to kill it
[19:26] * davidz (~Adium@ip68-96-75-123.oc.oc.cox.net) has joined #ceph
[19:28] <darkfaded> ah hehe
[19:29] <darkfaded> gregaf: i didnt rmeemb er that
[19:29] <darkfaded> it of course had died halfway
[19:29] <darkfaded> gregaf: who do i talk to if i want to make ceph-deploy work on non-deb distros and have patches?
[19:29] <darkfaded> because by now i would have been faster not using any automagic :)
[19:30] <gregaf> darkfaded: send them to the list; Sage or I will do something with them
[19:30] <darkfaded> kk
[19:31] * MikeMcClurg (~mike@firewall.ctxuk.citrix.com) Quit (Quit: Leaving.)
[19:33] * adjohn (~adjohn@ has joined #ceph
[19:39] <darkfaded> it is almost worky!
[19:42] * nwatkins (~Adium@soenat3.cse.ucsc.edu) has joined #ceph
[19:44] <nwatkins> There is a chunk of code in tools/ceph.cc for reading/writing admin sockets. This would be useful in unit tests to grab client perf counters. Would anyone be opposed to moving this to libcommon to avoid duplicating the functionality?
[19:45] <darkfaded> "mon.c@-1(probing) e0 discarding message auth(proto 0 26 bytes epoch 0) v1 and sending client elsewhere; we are not in quorum" - should I be seeing such messages when i believe i turned off anything related to cephx
[19:46] <gregaf> nwatkins: not sure if libcommon is the place for it or not (libcommon is huuuuge) but that's probably a better place for it than the ceph tool :)
[19:46] <gregaf> darkfaded: yes, those are sent even if the auth type is none
[19:46] <darkfaded> oh!
[19:47] <darkfaded> then something else is broken and not that
[19:47] <darkfaded> :=)
[19:47] <nwatkins> gregaf: hm, ok. how about something like a libcephtools
[19:47] <joao> darkfaded, by the looks of it, you don't have a formed quorum
[19:47] <darkfaded> there's 3 mons, they should manage
[19:47] <darkfaded> i'm telling them
[19:48] <joao> so what appears to be broken?
[19:48] <darkfaded> i simply don't know yet. everything "started" but a ceph -s will hang, as most other stuff. so connecting to mon is what seems to be broken
[19:49] <gregaf> nwatkins: dunno, splitting up libcommon, if appropriate, isn't something to do now
[19:49] <gregaf> that was just the first thought off the top of my head :)
[19:49] <joao> darkfaded, ceph -s hanging is usually a symptom of lack of quorum ;)
[19:50] <joao> darkfaded, try 'ceph --admin-daemon <path-to-one-of-the-admin-sockets> mon_status'
[19:50] <darkfaded> ok, and what is the most common cause for not forming a quorum?
[19:50] <darkfaded> ok sec
[19:50] <darkfaded> errr
[19:50] <darkfaded> and i never heard "admin socket" so far
[19:51] <darkfaded> is that the mon addr/port or something else?
[19:51] <joao> darkfaded, two out of three monitors not being up; or able to talk between themselves; or being unable to authenticate themselves to others (which could mean that you still have cephx enabled in a couple of them)
[19:51] <joao> darkfaded, it's a file somewhere down /var/
[19:51] * senner (~Wildcard@68-113-232-90.dhcp.stpt.wi.charter.com) has joined #ceph
[19:51] <joao> an .asok file
[19:51] <darkfaded> oki thanks
[19:51] * dspano (~dspano@rrcs-24-103-221-202.nys.biz.rr.com) has joined #ceph
[19:51] <joao> can't recall where it is kept
[19:51] * senner (~Wildcard@68-113-232-90.dhcp.stpt.wi.charter.com) Quit ()
[19:52] <gregaf> joao: darkfaded: that message means exactly that the monitor "is not in quorum"
[19:52] <joao> gregaf, yes
[19:52] <joao> it even states that as a reason for dropping the message :p
[19:53] <joao> but things could have evolved since darkfaded posted the log on the channel
[19:53] <gregaf> okay, you were just backing up to ceph -s hanging
[19:53] <gregaf> so I was confused
[19:53] <joao> yeah :)
[19:54] <joao> darkfaded, should be under /var/run/ceph/*.asok
[19:54] <darkfaded> thanks, finally found them there.
[19:56] <darkfaded> i think i found it and it would be funny
[19:56] <darkfaded> give me a second
[19:56] <darkfaded> and that command is lovely
[19:56] * Leseb (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) has joined #ceph
[19:56] <darkfaded> i guess thats where i should hook monitoring?
[19:56] <joao> it only gives you infos from the monitor you're hooking into
[19:58] <darkfaded> ok, i'll be comparing them too
[20:01] * Ryan_Lane (~Adium@ has joined #ceph
[20:04] * Tamil (~Adium@2607:f298:a:607:75c6:48e9:6b8a:3d50) has joined #ceph
[20:05] * nhorman (~nhorman@hmsreliant.think-freely.org) Quit (Quit: Leaving)
[20:05] * dmick (~dmick@2607:f298:a:607:7596:c46:d062:a53d) has joined #ceph
[20:07] <darkfaded> yay! *dance*
[20:08] <joao> :)
[20:09] <darkfaded> actually, funny. now the election worked and ceph -s cant find them any more due to some crazy error
[20:09] <darkfaded> tickle and kick
[20:09] <darkfaded> it
[20:11] <darkfaded> $gf fixed it.
[20:12] <darkfaded> i'll let her do the class next week
[20:13] <dmick> $gf lol
[20:32] * idnc_sk (~idnc_sk@ has joined #ceph
[20:32] <idnc_sk> hi
[20:32] <idnc_sk> ** ERROR: error creating empty object store in /ceph/osd/osd11: (22) Invalid argument
[20:33] <idnc_sk> any hints or should I start with log/conf gathering
[20:33] * calebmiles (~caleb@65-183-137-95-dhcp.burlingtontelecom.net) Quit (Ping timeout: 480 seconds)
[20:33] <idnc_sk> this err happens during mkcephfs, fs is xfs
[20:33] <idnc_sk> version :
[20:33] <idnc_sk> ceph version 0.49 (commit:ca6265d0f4d68a5eb82b5bfafb450e8e696633ac)
[20:34] <idnc_sk> krnl 3.5.7-gentoo
[20:35] <idnc_sk> === osd.11 ===
[20:35] <idnc_sk> 2012-11-30 20:27:57.539577 7f0d273b7780 -1 provided osd id 11 != superblock's -1
[20:35] <idnc_sk> 2012-11-30 20:27:57.541226 7f0d273b7780 -1 ** ERROR: error creating empty object store in /ceph/osd/osd11: (22) Invalid argument
[20:36] <idnc_sk> failed: '/usr/sbin/mkcephfs -d /tmp/mkcephfs.YhCE5s8FHE --init-daemon osd.11'
[20:36] <idnc_sk> >>> sorry for spam
[20:37] <darkfaded> joao, gregaf thanks again. gonna stop for today, the mons are running like a charm now and all osds are out. but i think after some beer this will be easier. i feel puzzled that i havent fixed it all by now :)
[20:37] <joshd> idnc_sk: was this from a second run of mkcephfs? it sounds like there was some existing data in /ceph/osd/osd11
[20:38] <idnc_sk> well, yes, I run mkfs first, then rechecked cfg(found an err in osd21 hostname) -> rerun again
[20:38] <joao> darkfaded, let us know if you run into any troubles :)
[20:38] <idnc_sk> ok, will clean-up osd and try again
[20:39] <idnc_sk> ou, and looking forward for the 0.55
[20:40] <idnc_sk> damn
[20:40] <idnc_sk> it worked!
[20:40] <idnc_sk> thx!
[20:41] <idnc_sk> now, the question is how good the gentoo-custom kernel will perform in a kvm virtio vm with hdd's attached from the host..
[20:42] <idnc_sk> ..with ceph on top -> btw managed to get a boottime of 2-4s for the vm :)
[20:43] <idnc_sk> on a q6700 8GB ram asus srv..
[20:53] * Tamil (~Adium@2607:f298:a:607:75c6:48e9:6b8a:3d50) Quit (Quit: Leaving.)
[20:55] * yasu` (~yasu`@dhcp-59-132.cse.ucsc.edu) has joined #ceph
[21:11] * Tamil (~Adium@ has joined #ceph
[21:19] * idnc_sk (~idnc_sk@ Quit (Quit: leaving)
[21:26] * idnc_sk (~idnc_sk@ has joined #ceph
[21:26] <idnc_sk> me again
[21:26] <idnc_sk> ..
[21:26] <idnc_sk> pg 0.1c2 is stuck creating, last acting []
[21:26] <idnc_sk> HEALTH_WARN 4224 pgs stuck inactive; 4224 pgs stuck unclean
[21:27] <idnc_sk> 4nodes, 2 osd's, 2mds's, 4 mons, fresh cephfs
[21:28] <joshd> can you pastebin 'ceph -s' and 'ceph osd tree'?
[21:28] <idnc_sk> yep, one sec
[21:30] <idnc_sk> http://pastebin.com/ztJMFs04
[21:30] * Tamil (~Adium@ Quit (Quit: Leaving.)
[21:31] <joshd> is 'ceph pg dump' showing all the acting and up sets as [] (i.e no osds)?
[21:32] <idnc_sk> ceph1 ~ # ceph pg dump | grep creating | wc -l
[21:32] <idnc_sk> 4224
[21:32] <idnc_sk> so - all pages stuck in creating state
[21:33] <idnc_sk> 11 557416 487589676 488147092 [] []
[21:33] <idnc_sk> 21 557416 487589676 488147092 [] []
[21:33] <idnc_sk> hoppa, forgot the headers
[21:33] <idnc_sk> osdstat kbused kbavail kb hb in hb out
[21:33] * roald (~Roald@ Quit (Read error: Connection reset by peer)
[21:34] * roald (~Roald@ has joined #ceph
[21:35] <idnc_sk> are these mount options for xfs OK
[21:35] <idnc_sk> rw,noexec,nodev,noatime,nodiratime,barrier=0
[21:35] <idnc_sk> ?
[21:35] <idnc_sk> well, the noexec part..
[21:36] <joshd> I think those should be fine
[21:36] * iggy__ (~iggy@theiggy.com) has joined #ceph
[21:37] * iggy (~iggy@theiggy.com) Quit (Read error: Connection reset by peer)
[21:38] <joshd> I suspect this is a bug fixed by later versions, but can't find a reference to it now
[21:39] <joshd> there was something like this fixed in 0.47, but you say you're running 0.49 everywhere?
[21:40] <idnc_sk> yep
[21:41] <idnc_sk> yep, came across a mailing list mentioning this
[21:41] <joshd> running 'ceph pg force_create_pg $pgid' for each pg should fix it up
[21:41] * loicd (~loic@jem75-2-82-233-234-24.fbx.proxad.net) Quit (Quit: Leaving.)
[21:41] <idnc_sk> hmm, ok, will try
[21:42] * iggy_ is now known as iggy
[21:42] <idnc_sk> ceph1 ~ # ceph pg force_create_pg 1.316
[21:42] <idnc_sk> pg 1.316 already creating
[21:42] <idnc_sk> tested with one..
[21:43] <idnc_sk> no go
[21:43] <idnc_sk> checked ceph logs/osd logs and dmesg, all looks fine
[21:44] <joshd> yeah, it seems the problem is the crushmap then
[21:45] <joshd> could you pastebin the decompiled crushmap? (ceph osd getcrushmap -o crushmap && crushtool -d crushmap)
[21:47] <idnc_sk> ooo, this is nice.. hold on, pasting
[21:49] <idnc_sk> http://pastebin.com/BkDN5m98
[21:50] <idnc_sk> btw does pastebin have an api so I could just curl the log and get a url back?
[21:50] * Tamil (~Adium@ has joined #ceph
[21:51] <idnc_sk> PastebinCL
[21:51] <joshd> there are command line tools that post to a bunch of pastbin sites
[21:52] <idnc_sk> those other devices in crushmap
[21:52] <idnc_sk> what are they?
[21:53] * lurbs_ (user@uber.geek.nz) has joined #ceph
[21:53] <joshd> they're not used, but they're the only thing strange I see there
[21:53] <idnc_sk> I'll paste the config, one sec
[21:54] * lurbs (user@uber.geek.nz) Quit (Read error: Connection reset by peer)
[21:56] <idnc_sk> http://pastebin.com/AeUnktx8
[21:59] <idnc_sk> 2n cluster, curr. 1xOsd per n, clnw(virtion to/throughg eth1 x-link) is on the todo list, currently virtio->vmbr(eth0)
[22:01] <joshd> is 0.49 just the latest available in gentoo?
[22:01] <idnc_sk> there is a 51 ebuild masked as unstable
[22:02] <idnc_sk> one sec
[22:03] <idnc_sk> yep
[22:03] <idnc_sk> http://packages.gentoo.org/package/sys-cluster/ceph
[22:03] <idnc_sk> but I can try the 51
[22:03] <idnc_sk> its a test setup
[22:04] * BManojlovic (~steki@ has joined #ceph
[22:05] <joshd> I suspect it'll be fixed by 0.51, or using osd.0 and osd.1 instead of osd.11 and osd.21
[22:05] <joshd> but 0.48.2 (latest stable release) or 0.54 (latest dev release) would be ideal
[22:06] <idnc_sk> well, if all goes well I plan to get rid of gluster so there will be 2 aditionall osd;s
[22:06] * miroslav (~miroslav@c-98-248-210-170.hsd1.ca.comcast.net) has joined #ceph
[22:08] <idnc_sk> ok plans a) 0.51 b) 054 c) osd rename(although, this worked on an earlier version)
[22:08] * idnc_sk @work..
[22:09] * mdawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[22:10] * slang (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) Quit (Remote host closed the connection)
[22:15] * slang (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) has joined #ceph
[22:20] * loicd (~loic@magenta.dachary.org) has joined #ceph
[22:43] * Tamil (~Adium@ Quit (Quit: Leaving.)
[22:45] * plut0 (~cory@pool-96-236-43-69.albyny.fios.verizon.net) has joined #ceph
[22:46] * Tamil (~Adium@ has joined #ceph
[23:00] * BManojlovic (~steki@ Quit (Remote host closed the connection)
[23:10] * BManojlovic (~steki@ has joined #ceph
[23:12] <idnc_sk> ..compiling 0.51
[23:15] * miroslav (~miroslav@c-98-248-210-170.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[23:20] * mib_4kh17m (17192e61@ircip3.mibbit.com) has joined #ceph
[23:25] * Tamil (~Adium@ Quit (Quit: Leaving.)
[23:29] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[23:29] * loicd (~loic@magenta.dachary.org) has joined #ceph
[23:32] * Tamil (~Adium@ has joined #ceph
[23:41] <nwatkins> Where is formatting done vector type when sent to ldout() ?
[23:45] * noob2 (~noob2@ext.cscinfo.com) Quit (Ping timeout: 480 seconds)
[23:49] * Tamil (~Adium@ Quit (Quit: Leaving.)
[23:55] * Tamil (~Adium@ has joined #ceph

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.