#ceph IRC Log

Index

IRC Log for 2013-01-28

Timestamps are in GMT/BST.

[0:10] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) has joined #ceph
[0:22] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[0:23] * loicd (~loic@magenta.dachary.org) has joined #ceph
[0:27] * The_Bishop_ (~bishop@e179020216.adsl.alicedsl.de) Quit (Quit: Wer zum Teufel ist dieser Peer? Wenn ich den erwische dann werde ich ihm mal die Verbindung resetten!)
[0:28] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) Quit (Ping timeout: 480 seconds)
[0:33] * gucki (~smuxi@84-73-202-7.dclient.hispeed.ch) Quit (Remote host closed the connection)
[0:36] * ScOut3R (~ScOut3R@dsl51B61EED.pool.t-online.hu) Quit (Ping timeout: 480 seconds)
[0:50] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) has joined #ceph
[0:51] * sagelap (~sage@182.255.120.124) has joined #ceph
[0:55] * The_Bishop (~bishop@e179020216.adsl.alicedsl.de) has joined #ceph
[1:08] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) Quit (Ping timeout: 480 seconds)
[1:11] * BillK (~BillK@124-148-195-207.dyn.iinet.net.au) Quit (Ping timeout: 480 seconds)
[1:17] * leseb_ (~leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) Quit (Remote host closed the connection)
[1:30] * BillK (~BillK@124-148-223-193.dyn.iinet.net.au) has joined #ceph
[1:31] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) has joined #ceph
[1:33] * xiaoxi (~xiaoxiche@134.134.137.71) has joined #ceph
[1:33] * psomas (~psomas@inferno.cc.ece.ntua.gr) Quit (Read error: Operation timed out)
[1:34] <xiaoxi> Hi, any known best practise for network tuning in OSD nodes?
[1:34] * psomas (~psomas@inferno.cc.ece.ntua.gr) has joined #ceph
[1:37] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) has joined #ceph
[1:37] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[1:39] * sagelap (~sage@182.255.120.124) Quit (Ping timeout: 480 seconds)
[1:49] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) Quit (Ping timeout: 480 seconds)
[2:03] * gohko (~gohko@natter.interq.or.jp) has joined #ceph
[2:12] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) has joined #ceph
[2:15] * LeaChim (~LeaChim@b01bdc44.bb.sky.com) Quit (Ping timeout: 480 seconds)
[2:18] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[2:20] * BillK (~BillK@124-148-223-193.dyn.iinet.net.au) Quit (Ping timeout: 480 seconds)
[2:30] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) Quit (Ping timeout: 480 seconds)
[2:38] * BillK (~BillK@124-169-127-113.dyn.iinet.net.au) has joined #ceph
[2:53] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) has joined #ceph
[3:03] <Pagefaulted> My osd hosts have 24 to 34 GB of ram for four osd deamons. Only 500gb data per deamon. Any way to put the extra memory to use?
[3:05] <phantomcircuit> HEALTH_WARN 15 pgs backfill; 15 pgs degraded; 16 pgs down; 95 pgs incomplete; 15 pgs recovering; 45 pgs stale; 95 pgs stuck inactive; 45 pgs stuck stale; 110 pgs stuck unclean; recovery 6623/183402 degraded (3.611%)
[3:05] <phantomcircuit> all 3 osd's are online but that hasn't changed in about an hour
[3:05] <phantomcircuit> Pagefaulted, it'll be used as read cache
[3:06] <Pagefaulted> Cool., thanks phantomcircuit
[3:07] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) Quit (Quit: Leaving.)
[3:11] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) Quit (Ping timeout: 480 seconds)
[3:29] * sagelap (~sage@182.255.120.124) has joined #ceph
[3:35] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) has joined #ceph
[3:47] * sagelap (~sage@182.255.120.124) Quit (Read error: No route to host)
[3:49] * sagelap (~sage@182.255.120.124) has joined #ceph
[3:53] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) Quit (Ping timeout: 480 seconds)
[3:57] * sagelap (~sage@182.255.120.124) Quit (Ping timeout: 480 seconds)
[4:16] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) has joined #ceph
[4:34] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) Quit (Ping timeout: 480 seconds)
[4:36] * miroslav (~miroslav@c-98-248-210-170.hsd1.ca.comcast.net) has joined #ceph
[4:51] * Ryan_Lane (~Adium@c-67-160-217-184.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[4:57] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) has joined #ceph
[5:08] * miroslav (~miroslav@c-98-248-210-170.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[5:15] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) Quit (Read error: Operation timed out)
[5:39] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) has joined #ceph
[5:44] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) has joined #ceph
[5:47] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) Quit ()
[5:48] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) has joined #ceph
[5:51] * stp (~stp@dslb-084-056-056-055.pools.arcor-ip.net) has joined #ceph
[5:55] * yoshi (~yoshi@p6124-ipngn1401marunouchi.tokyo.ocn.ne.jp) has joined #ceph
[5:56] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) Quit (Read error: Operation timed out)
[5:58] * scalability-junk (~stp@dslb-084-056-037-141.pools.arcor-ip.net) Quit (Ping timeout: 480 seconds)
[5:58] * nyeates (~nyeates@pool-173-59-239-231.bltmmd.fios.verizon.net) Quit (Quit: Zzzzzz)
[6:08] * Cube (~Cube@cpe-76-95-223-199.socal.res.rr.com) has joined #ceph
[6:13] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) Quit (Quit: Leaving.)
[6:20] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) has joined #ceph
[6:39] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) Quit (Ping timeout: 480 seconds)
[6:49] * sleinen (~Adium@217-162-132-182.dynamic.hispeed.ch) has joined #ceph
[6:50] * sleinen1 (~Adium@2001:620:0:26:1c97:8eea:46c2:ce2c) has joined #ceph
[6:57] * sleinen (~Adium@217-162-132-182.dynamic.hispeed.ch) Quit (Ping timeout: 480 seconds)
[7:02] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) has joined #ceph
[7:09] * sleinen1 (~Adium@2001:620:0:26:1c97:8eea:46c2:ce2c) Quit (Quit: Leaving.)
[7:21] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) Quit (Ping timeout: 480 seconds)
[7:43] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) has joined #ceph
[8:02] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) Quit (Ping timeout: 480 seconds)
[8:08] <xiaoxi> Hi, I sometimes get "heartbeat_map is_healthy 'OSD::op_tp thread 0x7f9eddba8700' had timed out after 15" during ceph running and some of the OSDs were marked down..after several minutes it's auto-recoveryed and everything looks fine.
[8:09] <xiaoxi> But I am pretty sure I have a stable network connection...
[8:15] * ctrl (~ctrl@95-24-95-24.broadband.corbina.ru) has joined #ceph
[8:24] * sjustlaptop (~sam@71-83-191-116.dhcp.gldl.ca.charter.com) Quit (Ping timeout: 480 seconds)
[8:25] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) has joined #ceph
[8:29] * gaveen (~gaveen@112.135.2.8) has joined #ceph
[8:31] <ctrl> Hello everyone!
[8:33] <ctrl> after restart ceph few days ago, cluster stuck in "HEALTH_WARN 2478 pgs degraded; 2496 pgs stuck unclean;"
[8:36] <ctrl> can someone tell me about, where i need search a problem?
[8:37] <ctrl> in osd logs i see this "fault with nothing to send, going to standby"
[8:39] * leseb (~leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) has joined #ceph
[8:42] * renzhi (~renzhi@116.226.50.112) has joined #ceph
[8:43] * loicd (~loic@3.46-14-84.ripe.coltfrance.com) has joined #ceph
[8:45] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) Quit (Ping timeout: 480 seconds)
[8:47] * leseb (~leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) Quit (Ping timeout: 480 seconds)
[8:56] <xiaoxi> ctrl:I also seen ""fault with nothing to send" often..I suppose it make no sense...
[8:56] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) has joined #ceph
[9:04] * madkiss (~madkiss@chello062178057005.20.11.vie.surfer.at) has joined #ceph
[9:04] * Vjarjadian (~IceChat77@5ad6d005.bb.sky.com) Quit (Quit: Always try to be modest, and be proud about it!)
[9:05] <ctrl> xiaoxi: good :) but now i don't have any idea how repair ceph
[9:08] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) has joined #ceph
[9:09] * loicd (~loic@3.46-14-84.ripe.coltfrance.com) Quit (Ping timeout: 480 seconds)
[9:14] * verwilst (~verwilst@dD5769628.access.telenet.be) has joined #ceph
[9:14] * BManojlovic (~steki@91.195.39.5) has joined #ceph
[9:17] * low (~low@188.165.111.2) has joined #ceph
[9:18] * loicd (~loic@3.46-14-84.ripe.coltfrance.com) has joined #ceph
[9:27] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) Quit (Ping timeout: 480 seconds)
[9:29] * leseb (~leseb@mx00.stone-it.com) has joined #ceph
[9:31] * wg (~g@ip98-177-222-249.ph.ph.cox.net) has joined #ceph
[9:33] <xiaoxi> 2013-01-28 16:33:13.247000 7fab8d7cb700 0 -- 192.101.11.124:6835/15438 >> 192.101.11.122:6801/53673 pipe(0x7d4c000 sd=27 :59559 s=1 pgs=4193 cs=14 l=1).connect claims to be 192.101.11.122:6801/54594 not 192.101.11.122:6801/53673 - wrong node!
[9:33] * Akendo (~akendo@p200300454F0792010290F5FFFEC8EEFA.dip.t-dialin.net) has joined #ceph
[9:33] <xiaoxi> I am seeing a lot of this log now, what does this log stand for ?
[9:34] * dpippenger (~riven@cpe-76-166-221-185.socal.res.rr.com) has joined #ceph
[9:36] * wg (~g@ip98-177-222-249.ph.ph.cox.net) Quit (Quit: Leaving)
[9:46] <ctrl> in which log u see this?
[9:48] * ScOut3R (~ScOut3R@212.96.47.215) has joined #ceph
[9:48] <xiaoxi> osd log
[9:49] <xiaoxi> how many ports do your osd listening on?
[9:49] <xiaoxi> In my setup ,a single osd daemon open 3 ports....seems strange
[9:50] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) has joined #ceph
[9:50] <xiaoxi> ceph-osd 16716 root 4u IPv4 71340 0t0 TCP 192.101.11.123:6800 (LISTEN)
[9:50] <xiaoxi> ceph-osd 16716 root 5u IPv4 3485296 0t0 TCP 192.101.11.123:6804 (LISTEN)
[9:50] <xiaoxi> ceph-osd 16716 root 6u IPv4 3485297 0t0 TCP 192.101.11.123:6805 (LISTEN)
[9:50] <ctrl> ports 6800-6820
[9:54] <xiaoxi> ceph's doc:open one port beginning at port 6800 for each OSD or metadata server that runs on the host.
[9:55] <xiaoxi> seems one port per osd is enough
[9:55] <xiaoxi> but why 3 ports are opened
[9:55] * Morg (d4438402@ircip1.mibbit.com) has joined #ceph
[9:58] * dosaboy (~user1@host86-164-229-186.range86-164.btcentralplus.com) has joined #ceph
[10:06] * xiaoxi (~xiaoxiche@134.134.137.71) Quit (Ping timeout: 480 seconds)
[10:08] * renzhi (~renzhi@116.226.50.112) Quit (Quit: Leaving)
[10:09] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) Quit (Ping timeout: 480 seconds)
[10:32] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) has joined #ceph
[10:42] * oldcrow (~x@c50-27-36-209.lbbkcmtk01.tx.dh.suddenlink.net) has joined #ceph
[10:50] * LeaChim (~LeaChim@b01bd420.bb.sky.com) has joined #ceph
[10:51] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) Quit (Ping timeout: 480 seconds)
[10:52] <jksM> ctrl, how many osds do you have?
[10:55] * jtangwk1 (~Adium@2001:770:10:500:75db:82f1:4ba1:c114) Quit (Quit: Leaving.)
[10:55] <ctrl> jksM: 12 osds
[10:55] <jksM> ctrl, on how many servers? are they all up and in?
[10:57] <ctrl> jksM: only 2, 1 mon, 2 mds, 12 osds they all up and in
[10:57] <jksM> have you set stuff with replication 3?
[10:58] <ctrl> yep
[10:58] * leseb (~leseb@mx00.stone-it.com) Quit (Read error: Connection reset by peer)
[10:58] <jksM> ctrl, probably you have set your crush map so that it can never succeed with only 2 servers - so that's the reason
[10:58] * leseb (~leseb@mx00.stone-it.com) has joined #ceph
[10:58] <jksM> ctrl, have you looked at "ceph osd tree" ?
[10:58] * jtangwk (~Adium@2001:770:10:500:d903:a9f1:e23:62f2) has joined #ceph
[10:59] * Anticimex (anticimex@netforce.csbnet.se) Quit (Quit: Changing server)
[10:59] <ctrl> -1 12 root default
[10:59] <ctrl> -3 12 rack unknownrack
[10:59] <ctrl> -2 6 host host01
[10:59] <ctrl> 0 1 osd.0 up 1
[10:59] <ctrl> 1 1 osd.1 up 1
[10:59] <ctrl> 2 1 osd.2 up 1
[10:59] <ctrl> 3 1 osd.3 up 1
[10:59] <ctrl> 4 1 osd.4 up 1
[10:59] <ctrl> 5 1 osd.5 up 1
[10:59] <ctrl> -4 6 host host02
[10:59] <ctrl> 20 1 osd.20 up 1
[10:59] <ctrl> 21 1 osd.21 up 1
[10:59] <ctrl> 22 1 osd.22 up 1
[10:59] <ctrl> 23 1 osd.23 up 1
[10:59] <ctrl> 24 1 osd.24 up 1
[11:00] <ctrl> 25 1 osd.25 up 1
[11:02] * Anticimex (anticimex@netforce.csbnet.se) has joined #ceph
[11:04] <jluis> morning #ceph
[11:05] * jluis is now known as joao
[11:08] <joao> ctrl, I'm not sure, but might your problem be because of the osd ids you assigned on host02? I know we are aiming at dissociating the osd id from its name, but don't know if we are there yet
[11:09] <joao> but then again, if they are up and in, and with those ids on the osdmap, that might not be the problem
[11:11] <jksM> ctrl: I don't know how you have your crush map set up, but if you have set it default and you only have 2 servers setup in the tree like that... you can never have HEALTH_OK with replication 3
[11:11] <jksM> ctrl, it will only have 2 servers to divide the data between, so how could it ever replicate it to 3 servers
[11:12] * stp (~stp@dslb-084-056-056-055.pools.arcor-ip.net) Quit (Ping timeout: 480 seconds)
[11:12] <joao> jksM, I'm pretty sure you could, but might be wrong? would you mind to elaborate on why?
[11:12] <ctrl> jksM: yes, i set replica to 3 because in near future want to add one more server
[11:13] <jksM> joao, you would need to configure ceph to specify that you think it is okay to have the same data be stored in multiple copies on the same server
[11:13] <jksM> joao, but by default it operates like that (as far as I know)
[11:13] <jksM> ctrl, okay, but then there's probably no problem at all... it just tells you that it hasn't reached replication 3 yet, because you haven't added the extra server yet
[11:14] <joao> the thing is, I can easily have replication 2 or 3 within a single server, with the default crush map
[11:14] <jksM> joao, oh okay, I'm probably wrong then
[11:14] <joao> just add the osds to the cluster, set replication to 3, and make sure you have 3 osds instead of 1 or 2
[11:14] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) has joined #ceph
[11:15] <joao> I'm sure you can mess that up with a wrong crushmap
[11:15] * gucki (~smuxi@80-218-125-247.dclient.hispeed.ch) has joined #ceph
[11:15] <gucki> hi there
[11:15] <jksM> gucki, hi
[11:16] <gucki> bobtail seems to be missing on the eu servers: http://ceph.com/debian-bobtail/ works, http://eu.ceph.com/debian-bobtail/ gives 404
[11:16] <joao> ctrl, could you post you 'pg dump' somewhere on pastebin or something?
[11:16] <ctrl> yeah, 1 min
[11:17] <joao> if that's something I have seen in the past, I might be able to understand what's happening
[11:17] <joao> gucki, poke wido :p
[11:17] <gucki> joao: what? ;-)
[11:18] <jtangwk> morning!
[11:18] <joao> wido is the one maintaining the eu mirror
[11:18] <joao> afaik
[11:18] <wido> hi
[11:18] <joao> morning wido :)
[11:18] <wido> I'll check. Might be that rsync didn't pick up the directory changes
[11:18] <wido> morning btw
[11:20] <gucki> joao: ah ok, thanks :()
[11:20] * musca (musca@tyrael.eu) has left #ceph
[11:23] <jksM> joao: but interesting, I need to read up on crush maps, as I thought it was actually difficult to get it to replicate multiple times on the same server
[11:23] * jksM is waiting for his ceph cluster to stabilise before testing can commence
[11:25] <ctrl> joao: http://pastebin.com/bKRuYAwc
[11:26] <joao> thanks, looking
[11:30] <joao> ctrl, would it be okay to try to set replication to 2 and see if the cluster gets healthy?
[11:30] <joao> as I see it, all your pgs are replicated across two osds, instead of 3, thus being degraded
[11:31] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) Quit (Read error: Operation timed out)
[11:32] <joao> ctrl, before you consider doing that, any chance you can post your crushmap somewhere?
[11:33] <joao> I forget the command, but should be something like 'ceph osd crush getmap -o crush.map' and 'crushtool -d crush.map'
[11:33] <ctrl> joao: already set replica to 2
[11:33] <joao> okay
[11:33] <joao> how's ceph -s?
[11:33] <ctrl> joao: about crushmap, yep, 1 min
[11:33] <ctrl> joao: HEALTH_WARN 2478 pgs degraded; 2496 pgs stuck unclean; recovery 11384/29495 degraded (38.596%)
[11:33] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) Quit (Ping timeout: 480 seconds)
[11:34] <joao> it's recovering; once that's finished we'll see what happens
[11:35] <joao> but from the pg dump, it appears as if all the pgs are replicated only over 2 osds, so there might be something on your crush map that jksM was right about (don't really know, but we'll see)
[11:36] <ctrl> joao: wait plz, i will post my crushmap
[11:37] <joao> I'll be here all day :)
[11:37] <jksM> joao, how are you using ceph? - in production? for qemu-kvm or?
[11:38] <ctrl> :)
[11:38] <joao> jksM, I'm on the dev team
[11:38] <jksM> ah, okay :)
[11:38] <joao> I don't really run it unless it's for testing
[11:39] <ctrl> joao: http://pastebin.com/s4Ru00hp
[11:42] <joao> ctrl, have to go afk for a little while; brb
[11:43] * lurpy (~test003@soho-94-143-249-78.sohonet.co.uk) Quit (Remote host closed the connection)
[11:47] * stp (~stp@dslb-084-056-056-055.pools.arcor-ip.net) has joined #ceph
[11:57] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) has joined #ceph
[12:01] * stp (~stp@dslb-084-056-056-055.pools.arcor-ip.net) Quit (Quit: Leaving)
[12:07] * ctrl (~ctrl@95-24-95-24.broadband.corbina.ru) Quit (Ping timeout: 480 seconds)
[12:07] * ctrl (~ctrl@89-178-200-4.broadband.corbina.ru) has joined #ceph
[12:13] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) has joined #ceph
[12:16] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) Quit (Ping timeout: 480 seconds)
[12:17] * ScOut3R_ (~ScOut3R@212.96.47.215) has joined #ceph
[12:19] * ScOut3R (~ScOut3R@212.96.47.215) Quit (Read error: Operation timed out)
[12:23] * match (~mrichar1@pcw3047.see.ed.ac.uk) has joined #ceph
[12:29] <ctrl> i'm back )
[12:31] <gucki> using cephx, is it possible to allow clients to read/write only a specific rbd image and disallow everything else?
[12:31] * jjgalvez (~jjgalvez@cpe-76-175-17-226.socal.res.rr.com) Quit (Ping timeout: 480 seconds)
[12:35] <gucki> and is there somewhere a list of all available class methods? :)
[12:39] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) has joined #ceph
[12:44] <joao> ctrl, how's 'ceph -s' looking like?
[12:51] * sagelap (~sage@diaman3.lnk.telstra.net) has joined #ceph
[12:54] <wido> gucki: No, that's not poissble
[12:54] <wido> what you'd have to do is place that RBD image in it's own pool and grant permissions accordingly
[12:54] <wido> But, be aware, you can't just create thousands of pools, that will hurt performance at some point
[12:55] <gucki> wido: ah ok, no that's not the way it want to go ...
[12:56] <gucki> wido: but is it only possible to grant r/w access to a rbd pool (so all existing images in it), but not allow creation/deletion/ etc of existing images?
[12:56] <wido> gucki: No, cephx is on RADOS level and that's 'below' RBD
[12:56] <wido> so when you grant r/w permission to a specific RADOS pool you can do whatever you want in that pool
[12:56] <gucki> wido: ok, too bad...but thanks for the info :)
[12:58] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) Quit (Ping timeout: 480 seconds)
[12:59] <ctrl> joao: like this HEALTH_WARN 2478 pgs degraded; 2496 pgs stuck unclean; recovery 11664/30050 degraded (38.815%)
[13:02] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) has joined #ceph
[13:05] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) Quit ()
[13:06] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) has joined #ceph
[13:07] * calebamiles (~caleb@c-107-3-1-145.hsd1.vt.comcast.net) has joined #ceph
[13:14] <joao> ctrl, right, just refreshed my crush knowledge, and the thing with with your crushmap and replication 3 is that the replication was being done on a host basis -- which having only two hosts wouldn't fully replicate
[13:15] <joao> however, if you really want to replicate over the all the osds, regardless of the host they're in, you could change the crushmap so that, on the rule buckets, you'd set 'osd' instead of 'host' on 'step choose'
[13:17] <joao> anyway, your cluster is still recovering from the replication change
[13:17] <joao> once that is finished, you should have a healthy cluster
[13:19] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) Quit (Quit: Leaving.)
[13:21] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) has joined #ceph
[13:23] <Kdecherf> Does anyone observed data corruption on cephfs/ceph-0.56.1?
[13:28] <ctrl> after change replica to 2, need ceph restart?
[13:30] * gregorg (~Greg@78.155.152.6) has joined #ceph
[13:37] <joao> ctrl, no
[13:39] <ctrl> joao: ok )
[13:40] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) Quit (Ping timeout: 480 seconds)
[13:43] * loicd (~loic@3.46-14-84.ripe.coltfrance.com) Quit (Ping timeout: 480 seconds)
[13:43] * coredumb (~coredumb@xxx.coredumb.net) Quit (Read error: Connection reset by peer)
[13:43] * scuttlemonkey (~scuttlemo@host86-189-11-203.range86-189.btcentralplus.com) has joined #ceph
[13:43] * ChanServ sets mode +o scuttlemonkey
[13:44] * coredumb (~coredumb@xxx.coredumb.net) has joined #ceph
[13:44] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) has joined #ceph
[13:49] * lx0 (~aoliva@lxo.user.oftc.net) has joined #ceph
[13:53] * Joel (~chatzilla@2001:620:0:46:f8f4:bd93:5a13:9f1f) has joined #ceph
[13:54] * The_Bishop_ (~bishop@e177091169.adsl.alicedsl.de) has joined #ceph
[13:55] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[13:57] * loicd (~loic@3.46-14-84.ripe.coltfrance.com) has joined #ceph
[13:59] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) Quit (Quit: Leaving.)
[14:00] * dosaboy (~user1@host86-164-229-186.range86-164.btcentralplus.com) Quit (Quit: Leaving.)
[14:00] * dosaboy (~user1@host86-164-229-186.range86-164.btcentralplus.com) has joined #ceph
[14:01] * The_Bishop (~bishop@e179020216.adsl.alicedsl.de) Quit (Ping timeout: 480 seconds)
[14:03] * noob2 (~noob2@pool-71-244-111-36.phlapa.fios.verizon.net) Quit (Quit: Leaving.)
[14:03] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) has joined #ceph
[14:08] * aliguori (~anthony@cpe-70-112-157-151.austin.res.rr.com) has joined #ceph
[14:13] * gaveen (~gaveen@112.135.2.8) Quit (Remote host closed the connection)
[14:22] * loicd (~loic@3.46-14-84.ripe.coltfrance.com) Quit (Ping timeout: 480 seconds)
[14:22] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) Quit (Ping timeout: 480 seconds)
[14:23] * nhorman (~nhorman@hmsreliant.think-freely.org) has joined #ceph
[14:30] <jmlowe> Kdecherf: corruption like this http://tracker.newdream.net/issues/3810 ?
[14:30] * scuttlemonkey (~scuttlemo@host86-189-11-203.range86-189.btcentralplus.com) Quit (Quit: This computer has gone to sleep)
[14:31] * loicd (~loic@3.46-14-84.ripe.coltfrance.com) has joined #ceph
[14:34] <Kdecherf> jmlowe: I don't use btrfs nor rdb atm
[14:36] <gucki> jmlowe: uh, this looks really bad. i'll stick with argoaut :-)
[14:38] * oldcrow (~x@c50-27-36-209.lbbkcmtk01.tx.dh.suddenlink.net) Quit (Quit: Computer has gone to sleep.)
[14:44] * leseb_ (~leseb@mx00.stone-it.com) has joined #ceph
[14:44] * leseb (~leseb@mx00.stone-it.com) Quit (Read error: Connection reset by peer)
[14:45] <jmlowe> gucki: It may just be some oddity of my setup, the dev's haven't been able to replicate
[14:48] * leseb (~leseb@mx00.stone-it.com) has joined #ceph
[14:48] * leseb_ (~leseb@mx00.stone-it.com) Quit (Read error: Connection reset by peer)
[14:49] <gucki> jmlowe: yeah i saw it's still open. but i'm always a little frightened when i read about corruption/ data loss. i'm really happy that i never had any data loss with argonaut, thought my cluster did go many bad situations (disks full, faulty disks, crashing nodes, ...) :)
[14:49] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) Quit (Ping timeout: 480 seconds)
[14:51] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) has joined #ceph
[14:51] <jmlowe> gucki: I have a slot on one of my arrays that silently corrupts data, took half a dozen different disks and a year to figure out
[14:51] * leseb_ (~leseb@mx00.stone-it.com) has joined #ceph
[14:51] * leseb (~leseb@mx00.stone-it.com) Quit (Read error: Connection reset by peer)
[14:51] * leseb_ is now known as leseb
[14:52] <gucki> jmlowe: mh, i always thought (hoped) filesystems have checksums to detect bad data...it really sucks when you search a bug in your code while it's in some underlying system. :(
[14:56] * jskinner (~jskinner@69.170.148.179) has joined #ceph
[14:57] * blufor (~blufor@adm-1.candycloud.eu) Quit (Ping timeout: 480 seconds)
[14:58] * jskinner (~jskinner@69.170.148.179) Quit (Remote host closed the connection)
[15:03] * jskinner (~jskinner@69.170.148.179) has joined #ceph
[15:04] * jskinner (~jskinner@69.170.148.179) Quit (Remote host closed the connection)
[15:10] * NightDog (~Karl@dhcp-009230.wlan.ntnu.no) has joined #ceph
[15:15] * Morg (d4438402@ircip1.mibbit.com) Quit (Quit: http://www.mibbit.com ajax IRC Client)
[15:21] * noob2 (~noob2@ext.cscinfo.com) has joined #ceph
[15:23] * low (~low@188.165.111.2) Quit (Quit: bbl)
[15:26] * NightDog (~Karl@dhcp-009230.wlan.ntnu.no) Quit (Quit: This computer has gone to sleep)
[15:37] * lx0 is now known as lxo
[15:47] * ctrl (~ctrl@89-178-200-4.broadband.corbina.ru) Quit ()
[15:53] * ctrl (~ctrl@89-178-200-4.broadband.corbina.ru) has joined #ceph
[15:59] * Akendo (~akendo@p200300454F0792010290F5FFFEC8EEFA.dip.t-dialin.net) Quit (Remote host closed the connection)
[16:00] * NightDog (~Karl@dhcp-009230.wlan.ntnu.no) has joined #ceph
[16:03] <Joel> Hi everybody, i am currently installing rados gateway and contnously get an http 403 error when trying the installation. The relevant pieces of the log-file can be found here: http://pastebin.com/ZyHKGzYa
[16:06] * xiaoxi (~xiaoxiche@134.134.139.72) has joined #ceph
[16:07] <xiaoxi> hi...reraise question:why my single osd daemon open 3 ports?
[16:08] <Kdecherf> xiaoxi: do you have mds and mon on this host?
[16:10] * loicd (~loic@3.46-14-84.ripe.coltfrance.com) Quit (Ping timeout: 480 seconds)
[16:10] <xiaoxi> I have a mon,but mon run on 6789
[16:10] <xiaoxi> osd opened 6800,6804 and 6805
[16:10] <joao> xiaoxi, one of them is for the monitor connection, another for inter-osd connections, and the other I'm not really sure
[16:11] * PerlStalker (~PerlStalk@72.166.192.70) has joined #ceph
[16:11] <joao> I see the ceph_osd.cc binding three times (client, cluster and heartbeat, thus I'm assuming the third should be for osd heartbeats)
[16:12] <joao> ah, no, the third is for client access, I think
[16:12] * mikedawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) has joined #ceph
[16:12] <xiaoxi> joao:thanks~
[16:13] <xiaoxi> have you got any idea about what this log meaning ?2013-01-28 16:33:13.247000 7fab8d7cb700 0 -- 192.101.11.124:6835/15438 >> 192.101.11.122:6801/53673 pipe(0x7d4c000 sd=27 :59559 s=1 pgs=4193 cs=14 l=1).connect claims to be 192.101.11.122:6801/54594 not 192.101.11.122:6801/53673 - wrong node!
[16:13] <xiaoxi> well, what are 54594 and 53673 stand for?
[16:14] <Kdecherf> random client port
[16:15] * NightDog (~Karl@dhcp-009230.wlan.ntnu.no) Quit (Quit: This computer has gone to sleep)
[16:15] <xiaoxi> but why it say wrong node?
[16:15] * ircolle (~ircolle@c-67-172-132-164.hsd1.co.comcast.net) has joined #ceph
[16:16] <joao> it's a nonce issue
[16:17] <joao> the nonce is taken from the daemon's pid
[16:17] <joao> so I guess you somehow changed the daemons pid (by restarting, maybe) and for some reason it is not aware of such thing
[16:18] <joao> which is in itself weird as hell
[16:19] * portante (~user@66.187.233.206) Quit (Quit: updating)
[16:19] <joao> <xiaoxi> well, what are 54594 and 53673 stand for? <- those are the nonces
[16:20] <joao> the address is made of ip:port/nonce (nonce being the pid)
[16:20] <xiaoxi> this log is quite often in my rack...
[16:21] <xiaoxi> sometimes this one:2013-01-28 06:57:24.513403 7f9ed0f14700 0 -- 192.101.11.123:6830/19742 >> 192.101.11.204:0/2412675826 pipe(0x4fd3400 sd=41 :6830 s=
[16:21] <xiaoxi> 0 pgs=0 cs=0 l=0).accept peer addr is really 192.101.11.204:0/2412675826 (socket is 192.101.11.204:55371/0)
[16:21] * sander (~chatzilla@108.196.205.12) has joined #ceph
[16:22] <xiaoxi> after seeing such logs, "slow request" and "heartbeat timeout" are likely to follow
[16:25] * loicd (~loic@3.46-14-84.ripe.coltfrance.com) has joined #ceph
[16:26] <xiaoxi> joao:I think 192.101.11.204:0/2412675826 is wrong ,right? port cannot be zero I think
[16:27] <joao> I'm assuming that's happening when you start the osd, is it not the case?
[16:27] <joao> oh, rather when that osd first receives a connection
[16:28] <joao> I'm not really sure what is going on there, I suppose that someone more familiar with the messenger and the osd should be able to help you better than I am
[16:28] <joao> maybe greg or sam when they come in
[16:29] <xiaoxi> no, my osd running for a long time (several days)~but this only happen when I doing some performance testing
[16:29] <joao> xiaoxi, are you by any chance assigning ports to the osds on ceph.conf?
[16:29] <joao> I recall some problems arising when that was done
[16:30] <xiaoxi> no..I just have public addr = 192.101.11.120
[16:30] <xiaoxi> cluster addr = 192.101.11.120
[16:30] <joao> okay
[16:31] <xiaoxi> maybe reboot all the daemons is the quickest way to bypass this issue,but not an ideal way
[16:32] <joao> I wouldn't advise that until someone who knows better say that's the best option
[16:33] <joao> specially if you're running that cluster in production
[16:34] <xiaoxi> yes, I recall I have reboot some of the daemons (1 or 2) to insert the debug option into it...is it the cause?
[16:35] <joao> highly unlikely
[16:35] <joao> debug usually only makes the daemons more verbose
[16:36] <joao> and that message has debug level 0, so you'd probably end up seeing it regardless of debugging being on a higher level or not
[16:36] <xiaoxi> I mean, reboot changed the daemons pid as you said befor
[16:36] <joao> so I guess seeing that message is not a side-effect of higher debug levels
[16:36] <joao> ah
[16:36] <joao> still doubt it
[16:36] <xiaoxi> doubt it too :)...
[16:36] <joao> but don't know
[16:37] <joao> let me check if this is on the tracker (should have done this first but totally forgot!)
[16:38] <joao> btw, I'm assuming that log comes from the osd, is that correct?
[16:40] <joao> xiaoxi, which version are you using?
[16:40] <xiaoxi> yes,from osd
[16:41] <xiaoxi> I am using the daily build rpm of main brunch
[16:41] <xiaoxi> ceph version 0.56-417-g67c7757 (67c77577bdfe4985aa50e91986677c742b7cc85f)
[16:43] <joao> the 'wrong node' one looks an awful lot like http://tracker.newdream.net/issues/3548
[16:43] <joao> which has been marked resolved a month ago
[16:43] <joao> (almost two months really)
[16:46] <jtangwk> hmm that change in port numbers for ceph mon caught me out today
[16:46] * vata (~vata@2607:fad8:4:6:452d:e2d2:9d03:69b9) has joined #ceph
[16:47] <joao> and the last one might be a monitor issue instead of an osd issue, as there were a couple of issues on that regard a year ago (also resolved, but something might have crept in in the meantime)
[16:47] <joao> jtangwk, what has?
[16:47] <xiaoxi> joao: so it should be in the main brunch for a long time.
[16:48] <joao> xiaoxi, I don't know; we should run this by someone else
[16:50] <xiaoxi> yes, I would like to try to remove the debug options(since it producing a lot of logs..which happen to share the same disk with mon's data) to see if it is getting better...
[16:50] <jtangwk> the default port was 6789 in argonaut
[16:50] <jtangwk> but its 9089 in bobtail from what i can tell
[16:51] <xiaoxi> Is it a typical deployment to move monitor in one subnet while leaving osd for another subnet?
[16:51] <jtangwk> i had some scripts watching some ports to restart services in my test setup
[16:51] <joao> jtangwk, might have changed, but that's the first time and hearing about that
[16:51] <joao> let me check
[16:53] <jtangwk> never mind
[16:53] <jtangwk> it was my stupidity
[16:53] <jtangwk> it was a process id that was printed and not a port number
[16:53] <jtangwk> :P
[16:53] <joao> ah
[16:54] <joao> good thing I only had time to open gitk before freaking out
[16:54] <jtangwk> heh
[16:54] <jtangwk> been ansibling up a chunk of my test infrastructure
[16:57] * match (~mrichar1@pcw3047.see.ed.ac.uk) Quit (Quit: Leaving.)
[16:57] * loicd (~loic@3.46-14-84.ripe.coltfrance.com) Quit (Quit: Leaving.)
[17:01] * scalability-junk (~stp@188-193-201-35-dynip.superkabel.de) has joined #ceph
[17:01] * sander (~chatzilla@108.196.205.12) Quit (Quit: ChatZilla 0.9.89 [Firefox 17.0.1/20121128204232])
[17:08] * Vjarjadian (~IceChat77@5ad6d005.bb.sky.com) has joined #ceph
[17:08] * Joel (~chatzilla@2001:620:0:46:f8f4:bd93:5a13:9f1f) Quit (Quit: ChatZilla 0.9.89 [Firefox 18.0.1/20130116073211])
[17:09] * BillK (~BillK@124-169-127-113.dyn.iinet.net.au) Quit (Ping timeout: 480 seconds)
[17:14] * xiaoxi (~xiaoxiche@134.134.139.72) Quit (Remote host closed the connection)
[17:21] * NightDog (~Karl@dhcp-009230.wlan.ntnu.no) has joined #ceph
[17:22] * BillK (~BillK@58-7-146-102.dyn.iinet.net.au) has joined #ceph
[17:23] * NightDog (~Karl@dhcp-009230.wlan.ntnu.no) Quit ()
[17:27] * gaveen (~gaveen@112.135.0.167) has joined #ceph
[17:28] * jlogan (~Thunderbi@2600:c00:3010:1:986b:ae1d:420:6c44) has joined #ceph
[17:31] * jlogan (~Thunderbi@2600:c00:3010:1:986b:ae1d:420:6c44) Quit ()
[17:37] * jlogan (~Thunderbi@2600:c00:3010:1:5c67:1323:2b43:da43) has joined #ceph
[17:44] * NightDog (~Karl@dhcp-009230.wlan.ntnu.no) has joined #ceph
[17:45] * loicd (~loic@magenta.dachary.org) has joined #ceph
[17:46] * ScOut3R_ (~ScOut3R@212.96.47.215) Quit (Ping timeout: 480 seconds)
[17:51] * sstan (~chatzilla@dmzgw2.cbnco.com) Quit (Remote host closed the connection)
[17:55] * scalability-junk (~stp@188-193-201-35-dynip.superkabel.de) Quit (Quit: Leaving)
[17:56] * scalability-junk (~stp@188-193-201-35-dynip.superkabel.de) has joined #ceph
[17:57] * NightDog (~Karl@dhcp-009230.wlan.ntnu.no) Quit (Quit: This computer has gone to sleep)
[17:59] * drokita (~drokita@199.255.228.128) has joined #ceph
[18:03] <drokita> My pg distributions are getting out of whack again. From what I am hearing this is because my pgs are not in a power of 2. Is this correct?
[18:03] * scuttlemonkey (~scuttlemo@217.39.7.254) has joined #ceph
[18:03] * ChanServ sets mode +o scuttlemonkey
[18:03] <drokita> I have deltas as high as 30% between the highest utilized and lowest utilized OSD
[18:05] * xmltok (~xmltok@cpe-76-170-26-114.socal.res.rr.com) Quit (Remote host closed the connection)
[18:05] * xmltok (~xmltok@pool101.bizrate.com) has joined #ceph
[18:05] <iggy> drokita: how many PGs do you have?\
[18:05] <gucki> drokita: i see the same on my cluster, would be nice to know :)
[18:05] <noob2> i see the same on my cluster for a few drives
[18:07] <drokita> I have 24 OSDS, 4 servers (10 each) and 4608 pgs
[18:07] <drokita> I am using my system for backup. It is very problematic for me because if I rebalance, it disrupts my ability to hit my backup windows.
[18:08] <drokita> I also have a cluster that has 40 OSDs, 4 servers and 4224 pgs.... it has the same problem, but since the storage is higher it doesn't peak as fast
[18:08] * topro (~topro@host-62-245-142-50.customer.m-online.net) Quit (Read error: Connection reset by peer)
[18:10] * gregaf (~Adium@cpe-76-174-249-52.socal.res.rr.com) has joined #ceph
[18:11] * BillK (~BillK@58-7-146-102.dyn.iinet.net.au) Quit (Ping timeout: 480 seconds)
[18:12] <noob2> i haven't tried a rebalance yet but i'm assuming it would kill my cluster performance for a few hours
[18:13] <drokita> In argonaut, it doesn't ***kill*** performance, but it does have enough of an impact that it becomes problematic
[18:13] <noob2> i'm on the bobtail branch
[18:14] <noob2> maybe i'm being dramatic here. it doesn't kill it but there's a big slow down
[18:15] <drokita> Really... Bobtail is supposed to fix the performance issues during rebalance. I have a lot of expectations set that it will
[18:15] <gregaf> bobtail works better across a wider range of configurations than argonaut did and is smarter about how much it devotes to recovery
[18:16] <gregaf> but if you have specific expectations you still need to tune it to match them
[18:16] <drokita> what do you mean by 'tune it'? Tune the cluster, or tune my expectations?
[18:16] <drokita> or both :)
[18:17] * ScOut3R (~ScOut3R@2E6BA0D4.dsl.pool.telekom.hu) has joined #ceph
[18:17] <gregaf> well, I meant the first, but both works even better ;)
[18:17] <noob2> lol
[18:17] * dosaboy (~user1@host86-164-229-186.range86-164.btcentralplus.com) Quit (Remote host closed the connection)
[18:18] * dosaboy (~user1@host86-164-229-186.range86-164.btcentralplus.com) has joined #ceph
[18:19] <drokita> Is anyone aware of a guide that can be used for tuning it. Our production cluster is about 2 years old. It was built to recommendations at the time, but it seems once configured it is hard to change down the road.
[18:23] <gregaf> they're normal configurables that you can inject or stick in the ceph.conf and restart daemons
[18:23] <gregaf> I don't have them in my head though and I don't think we have a doc on recovery tuning yet
[18:24] <drokita> We have tried to set the number of recovery process down from 5 to 1. It doesn't seem to have any effect on real world performance.
[18:24] <gregaf> it does in Bobtail :)
[18:25] * benr (~benr@puma-mxisp.mxtelecom.com) has joined #ceph
[18:25] <drokita> Cool. I am looking forward to it then.
[18:26] <drokita> The Inktank is in the process of gathering some upgrade doc for us to make that jump
[18:28] * BillK (~BillK@124-169-55-99.dyn.iinet.net.au) has joined #ceph
[18:28] <joao> gregaf, have a minute?
[18:28] <gregaf> sure
[18:29] <joao> xiaoxi was having some 'wrong node' issues on the osd earlier today, and another regarding a 'peer addr is really some other'
[18:29] * sstan (~chatzilla@dmzgw2.cbnco.com) has joined #ceph
[18:30] <joao> by looking in the tracker it looked like both had been resolved sometime in the past (one of them was a year old)
[18:30] <joao> not sure if this was something you'd be familiar with?
[18:30] * alram (~alram@38.122.20.226) has joined #ceph
[18:30] <gregaf> I think it's usually a result of cluster flapping
[18:30] <gregaf> I skimmed a bit of the log; was he the one who also had a bunch of op_tp timeouts?
[18:32] <joao> can't recall, let me check
[18:32] * avoine (~avoine@modemcable121.95-163-184.mc.videotron.ca) has joined #ceph
[18:32] <joao> yep
[18:32] <gregaf> yeah, that'll be it then
[18:33] <gregaf> especially with Sage's new heartbeat code which I think he must have had for the timeout after 15, which references the state of the op threads when doing network heartbeats
[18:33] <gregaf> joao: while you're around, do you remember which monitor/map has a set of cluster features?
[18:34] <gregaf> I need to add one to the MDSMap and would love a reference ;)
[18:35] <joao> I believe they are all stashed on 'feature_set', directly on the store
[18:35] <joao> let me check
[18:35] <gregaf> ah, and it's only the monitors, okay
[18:35] <gregaf> I thought one of the maps included one as well, but it looks like not
[18:36] <joao> they are shared sometime during election, I think
[18:36] <joao> but they are kept by the monitors separately
[18:37] <gregaf> okay, not a lot of help then :(
[18:37] <gregaf> thanks!
[18:37] * gucki (~smuxi@80-218-125-247.dclient.hispeed.ch) Quit (Ping timeout: 480 seconds)
[18:38] * NightDog (~Karl@38.179.202.84.customer.cdi.no) has joined #ceph
[18:38] <joao> gregaf, you might want to look into 'check_features()' and 'get_supported_features()' on Monitor.cc
[18:38] <gregaf> danke
[18:39] * NightDog (~Karl@38.179.202.84.customer.cdi.no) Quit (Read error: Connection reset by peer)
[18:40] <joao> let me know if there's anything I can assist you on
[18:41] * NightDog (~Karl@38.179.202.84.customer.cdi.no) has joined #ceph
[18:41] * NightDog (~Karl@38.179.202.84.customer.cdi.no) Quit (Read error: Connection reset by peer)
[18:43] <gregaf> nope, was just hoping for a model to make it go a little faster; it's nothing tricky either way
[18:43] * NightDog (~Karl@38.179.202.84.customer.cdi.no) has joined #ceph
[18:43] * BManojlovic (~steki@91.195.39.5) Quit (Remote host closed the connection)
[18:44] * NightDog (~Karl@38.179.202.84.customer.cdi.no) Quit (Read error: Connection reset by peer)
[18:47] * NightDog (~Karl@38.179.202.84.customer.cdi.no) has joined #ceph
[18:48] * NightDog (~Karl@38.179.202.84.customer.cdi.no) Quit (Read error: Connection reset by peer)
[18:50] * NightDog (~Karl@38.179.202.84.customer.cdi.no) has joined #ceph
[18:50] * xmltok (~xmltok@pool101.bizrate.com) Quit (Quit: Leaving...)
[18:50] * NightDog (~Karl@38.179.202.84.customer.cdi.no) Quit (Read error: Connection reset by peer)
[18:50] * leseb (~leseb@mx00.stone-it.com) Quit (Remote host closed the connection)
[18:51] * NightDog (~Karl@38.179.202.84.customer.cdi.no) has joined #ceph
[18:51] * avoine (~avoine@modemcable121.95-163-184.mc.videotron.ca) Quit (Ping timeout: 480 seconds)
[18:52] * NightDog (~Karl@38.179.202.84.customer.cdi.no) Quit (Read error: Connection reset by peer)
[18:53] * NightDog (~Karl@38.179.202.84.customer.cdi.no) has joined #ceph
[18:54] * NightDog (~Karl@38.179.202.84.customer.cdi.no) Quit (Read error: Connection reset by peer)
[18:55] * NightDog (~Karl@38.179.202.84.customer.cdi.no) has joined #ceph
[18:55] * NightDog (~Karl@38.179.202.84.customer.cdi.no) Quit (Read error: Connection reset by peer)
[19:00] * avoine (~avoine@modemcable121.95-163-184.mc.videotron.ca) has joined #ceph
[19:00] <avoine> hello, I'm trying to make radosgw working with keystone but I have problems getting the key format right in radosgw keyring
[19:01] * chutzpah (~chutz@199.21.234.7) has joined #ceph
[19:02] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[19:02] * loicd (~loic@magenta.dachary.org) has joined #ceph
[19:02] <avoine> I'm thinking its that because I have this error in the log: -12> 2013-01-25 16:21:19.603989 afffab40 0 ERROR: signer 0 status = SigningCertNotFound
[19:02] * gucki (~smuxi@84-72-9-205.dclient.hispeed.ch) has joined #ceph
[19:02] <avoine> after: sending request to http://127.0.0.1:35357/v2.0/tokens/revoked
[19:03] <avoine> I'm using version 0.56.1-0ubuntu1
[19:04] * gucki (~smuxi@84-72-9-205.dclient.hispeed.ch) Quit (Remote host closed the connection)
[19:09] * Vjarjadian (~IceChat77@5ad6d005.bb.sky.com) Quit (Quit: IceChat - Keeping PC's cool since 2000)
[19:10] <avoine> also I do not use nss
[19:11] * dosaboy (~user1@host86-164-229-186.range86-164.btcentralplus.com) Quit (Remote host closed the connection)
[19:11] * dosaboy (~user1@host86-164-229-186.range86-164.btcentralplus.com) has joined #ceph
[19:13] * sjustlaptop (~sam@2607:f298:a:697:f4bf:4b06:6be1:7013) has joined #ceph
[19:16] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) Quit (Quit: Leaving.)
[19:17] * sjustlaptop (~sam@2607:f298:a:697:f4bf:4b06:6be1:7013) Quit (Remote host closed the connection)
[19:17] * BillK (~BillK@124-169-55-99.dyn.iinet.net.au) Quit (Ping timeout: 480 seconds)
[19:17] * amichel (~amichel@salty.uits.arizona.edu) has joined #ceph
[19:18] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) has joined #ceph
[19:19] <amichel> So, I'm making a new rbd device and a vagary I hadn't noticed occured to me. There's -p, --pool for source pool but also --dest-pool. What is a destination pool vs a source pool?
[19:21] * xmltok (~xmltok@pool101.bizrate.com) has joined #ceph
[19:21] * gregaf (~Adium@cpe-76-174-249-52.socal.res.rr.com) Quit (Quit: Leaving.)
[19:28] * gregaf (~Adium@cpe-76-174-249-52.socal.res.rr.com) has joined #ceph
[19:30] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) has joined #ceph
[19:31] * Ryan_Lane (~Adium@c-67-160-217-184.hsd1.ca.comcast.net) has joined #ceph
[19:32] <noob2> :D
[19:33] * Tamil1 (~tamil@38.122.20.226) has joined #ceph
[19:35] <joshd> amichel: --dest-pool is just for things like rbd cp or rbd clone, where source and destination could differ
[19:35] <amichel> Oh, I see
[19:35] <amichel> Well, with that in mind, I think the first example in "Creating a block device image" probably needs to be fixed: http://ceph.com/docs/master/rbd/rados-rbd-cmds/
[19:36] <amichel> It has --dest-pool for a create parameter which did not do what you'd think it does in that context :D
[19:37] <joshd> ah, thanks for pointing that out
[19:42] * miroslav (~miroslav@173-228-38-131.dsl.dynamic.sonic.net) has joined #ceph
[19:43] * dosaboy (~user1@host86-164-229-186.range86-164.btcentralplus.com) Quit (Quit: Leaving.)
[19:46] * gregaf (~Adium@cpe-76-174-249-52.socal.res.rr.com) Quit (Quit: Leaving.)
[19:47] * Cube (~Cube@cpe-76-95-223-199.socal.res.rr.com) Quit (Quit: Leaving.)
[19:47] * gregaf (~Adium@cpe-76-174-249-52.socal.res.rr.com) has joined #ceph
[19:49] * dosaboy (~user1@host86-164-229-186.range86-164.btcentralplus.com) has joined #ceph
[19:50] * BManojlovic (~steki@46-172-222-85.adsl.verat.net) has joined #ceph
[19:50] * portante (~user@66.187.233.206) has joined #ceph
[19:54] <amichel> Oh sure, no problem
[19:55] * Tamil1 (~tamil@38.122.20.226) Quit (Quit: Leaving.)
[19:55] <amichel> Anyone have any guidance about best-practice filesystems to use on rbd devices? I know any filesystem will work, but I was wondering if there was one that was better or more tunable for the unique characteristics of the rbd layer?
[19:57] * gaveen (~gaveen@112.135.0.167) Quit (Remote host closed the connection)
[19:58] <wer> I have a weird thing where pulling the entire list of objects using python/boto says I have 3.3 millionish. boto is getting duplicate keys and I don't know why.
[19:58] <wer> I am sure it must be my code but damn.
[20:04] <jmlowe> does anybody know how to interpret the filestore debugging logs?
[20:06] * Cube1 (~Cube@12.248.40.138) has joined #ceph
[20:22] * dmick (~dmick@2607:f298:a:607:8996:57cd:5193:a7c0) has joined #ceph
[20:25] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[20:26] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[20:28] * cbm64 (4f130076@ircip1.mibbit.com) has joined #ceph
[20:28] * verwilst (~verwilst@dD5769628.access.telenet.be) Quit (Quit: Ex-Chat)
[20:33] <sjust> jmlowe: what's up?
[20:37] <cbm64> hi every1!
[20:37] <cbm64> is anyone out there?
[20:38] * alram (~alram@38.122.20.226) Quit (Quit: leaving)
[20:39] <dmick> cbm64: lots of us
[20:40] <cbm64> ok, warning... ceph newbie here
[20:40] <dmick> how can we help
[20:40] <dmick> ?
[20:42] <cbm64> tnx in advance... ceph cluster setup ok, 1 mon 6 osds 1 mds running ok on 2 machines. rdb ok cephfs ok . Added one more osd on a third machine, and lot of "pgs stuck"...
[20:42] <jmlowe> sjust: I thew up some more logs for you on 3810
[20:43] <jmlowe> sjust: couldn't really make heads or tails of the entries like this "write 2.1fe_head/1a941ffe/rb.0.1329.238e1f29.0000000017bf/head//2 2346496~4096 = 4096"
[20:43] <dmick> cbm64: ceph osd tree output (probably to a pastebin) is a good first question
[20:43] <sjust> the write to that object of 4096 at offset 0 returned 4096
[20:44] <sjust> oops, offset 2346496
[20:44] <dmick> "offset~len" is a convention in the logs
[20:44] <sjust> right
[20:46] <jmlowe> huh, so add up all of the len entries and get the total amount written provided nothing is written twice
[20:46] * bletch (c1cd3c85@ircip1.mibbit.com) has joined #ceph
[20:46] <bletch> http://pastebin.com/HmrxhREi
[20:47] <sjust> sorry?
[20:47] <dmick> is bletch == cbm64?
[20:47] * cbm64-1 (c1cd3c85@ircip1.mibbit.com) has joined #ceph
[20:47] <cbm64-1> http://pastebin.com/HmrxhREi
[20:47] <dmick> == cbm64-1?
[20:48] <dmick> I feel like we're at a masked ball
[20:48] <cbm64-1> mibbit disconnected me (cbm64) and forced to use another nickname dmick !
[20:48] <dmick> ok so tree shows that osd.6 isn't properly added to the hierarchy
[20:48] <cbm64-1> uh oh...
[20:48] <dmick> what did you do while adding the osd to get it added to the crush map?
[20:48] * cbm64 (4f130076@ircip1.mibbit.com) Quit (Quit: http://www.mibbit.com ajax IRC Client)
[20:51] <dmick> ^cbm64-1
[20:51] <cbm64-1> followed http://ceph.com/docs/master/rados/operations/add-or-rm-osds/ with a correction "root=default" suggested in a previous irc http://irclogs.ceph.widodh.nl/index.php?date=2012-12-11
[20:53] <dmick> do you still have the exact crush add command handy?
[20:53] <cbm64-1> because ceph osd crush set {id-or-name} {weight} pool={pool-name} gave me an error "invalid argument (22)"
[20:54] <cbm64-1> ceph osd crush set 6 osd.6 2.0 root=default (exact command taken from bash history)
[20:54] * avoine (~avoine@modemcable121.95-163-184.mc.videotron.ca) Quit (Read error: Operation timed out)
[20:55] <dmick> so, two things:
[20:55] <dmick> 1) you've supplied both id and name, which ought to have been a syntax error
[20:55] <dmick> 2) supplying only "root=default" means that you've asked for it to be put in the hierarchy at the root, default, which is probably not where you want it
[20:56] <dmick> trying to add my own to see what happens with both id and name
[20:56] <joshd> dmick: 1) is just the old syntax
[20:56] <dmick> ah, that would explain it; which version are you running cbm64-1?
[20:57] * avoine (~avoine@modemcable121.95-163-184.mc.videotron.ca) has joined #ceph
[20:57] <cbm64-1> ceph version 0.56.1 (e4a541624df62ef353e754391cbbb707f54b16f7) kernel 3.6.11 onto 12.04 ubuntu box
[20:58] * ctrl (~ctrl@89-178-200-4.broadband.corbina.ru) Quit (Ping timeout: 480 seconds)
[20:59] * ctrl (~ctrl@95-24-247-139.broadband.corbina.ru) has joined #ceph
[20:59] <jmlowe> sjust: so if the largest offset write starts at 4185600 and is 8704 how is it that du -ab /data/osd.3/current/2.1fe_head/DIR_E/DIR_F/DIR_F/rb.0.1329.238e1f29.0000000017bf__head_1A941FFE__2
[20:59] <jmlowe> 4186112 /data/osd.3/current/2.1fe_head/DIR_E/DIR_F/DIR_F/rb.0.1329.238e1f29.0000000017bf__head_1A941FFE__2
[20:59] <dmick> cbm64-1: is the new osd on host sto1?
[21:00] <elder> back in a bit
[21:00] <cbm64-1> the new osd (osd.6) in on host os0
[21:01] <dmick> is this a host that is not yet in the crush map?
[21:02] <cbm64-1> dont know sorry... which cmd i should use to see that?
[21:02] * ScOut3R (~ScOut3R@2E6BA0D4.dsl.pool.telekom.hu) Quit (Remote host closed the connection)
[21:02] <dmick> osd tree shows two hosts, sto0 and sto1
[21:02] <dmick> really I'm just asking if you're adding a new host
[21:03] <cbm64-1> sto0 contains osd.[0-3] sto1 contains osd.[4-5] os0 contains osd.6
[21:03] <dmick> so yes, it's a new host
[21:04] <cbm64-1> os0 was the last added
[21:04] <dmick> that's all I was after
[21:05] <cbm64-1> the very first ceph cluster setup included only sto[0-1] and worked great
[21:05] <dmick> so probably you want something like "ceph osd crush set 6 osd.6 2.0 root=default rack=unknownrack host=os0"
[21:06] <cbm64-1> yes... maybe it should be as you suggest... could i recover from this situation?
[21:06] <dmick> an alternative is to extract the crush map, decompile it, edit the new OSD in by hand, compile and reinject, as shown at http://ceph.com/docs/master/rados/operations/crush-map/#editing-a-crush-map
[21:06] <dmick> but this crush set is easier
[21:07] <dmick> you should be able to do either now and the cluster will take the new map and operate on it
[21:07] * leseb (~leseb@5ED17881.cm-7-2b.dynamic.ziggo.nl) has joined #ceph
[21:07] <jmlowe> sjust: as far as I can tell there is a write recorded in my osd logs that didn't make it to the disk on the secondary
[21:07] <cbm64-1> ok, so can I give "ceph osd crush set 6 osd.6 2.0 root=default rack=unknownrack host=os0" command just now on sto0?
[21:08] <jmlowe> sjust: exactly 8k worth
[21:08] <dmick> on any host that can contact the monitors, which should be any; sto0 is fine
[21:08] <cbm64-1> well i'm trying just now....
[21:10] <cbm64-1> dmick HEALTH_WARN 76 pgs backfill; 18 pgs recovering; 197 pgs recovery_wait; 291 pgs stuck unclean; recovery 10295/23427 degraded (43.945%)
[21:10] <dmick> backfill/recovery are filling the new osd
[21:10] <dmick> check out ceph osd tree again
[21:11] <cbm64-1> dmick http://pastebin.com/h1BGL5dH
[21:12] <dmick> that looks better
[21:12] <dmick> check ceph -s periodically, or watch ceph -w, and see if the cluster doesn't become healthy
[21:12] <cbm64-1> yes, it look really better ! HEALTH_WARN 75 pgs backfill; 20 pgs recovering; 146 pgs recovery_wait; 241 pgs stuck unclean; recovery 8454/23380 degraded (36.159%)
[21:13] <cbm64-1> degraded percentage decreases !!! THANKS A LOT dmick!
[21:14] <dmick> not happy about the 241 stuck unclean, but let's wait a bit and let backfill finish
[21:15] <cbm64-1> 191 pgs stuck unclean now... also decreasing..!
[21:16] <dmick> that's good. I wasn't sure about that, but suspected.
[21:16] <dmick> so for a clue why this went wrong, decompile your crush map to source as the link about editing above shows you
[21:16] <dmick> and paste it
[21:17] <cbm64-1> i dont know how to decompile crush map... very sorry. total newbie :(
[21:17] <dmick> (12:06:45 PM) dmick: an alternative is to extract the crush map, decompile it, edit the new OSD in by hand, compile and reinject, as shown at http://ceph.com/docs/master/rados/operations/crush-map/#editing-a-crush-map
[21:18] <dmick> yjsy
[21:18] <dmick> that's what I meant by "link about editing"
[21:19] * nhm (~nh@184-97-251-146.mpls.qwest.net) has joined #ceph
[21:20] <cbm64-1> ok i think that have to study on decompiling crush map reading docs as you suggested...
[21:21] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) Quit (Quit: Leaving.)
[21:21] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) has joined #ceph
[21:22] * sleinen (~Adium@2001:620:0:26:d911:f3c9:6bf3:784b) has joined #ceph
[21:23] <dmick> cbm64-1: it's just two commands
[21:23] <cbm64-1> done... http://pastebin.com/Km3P8bFz
[21:24] <dmick> ceph osd getcrushmap -o /tmp/map ; crushtool -d map.txt -o /tmp/map
[21:24] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[21:24] <dmick> so as you can see there's a lot of stuff there generated for you
[21:25] <dmick> when Ceph wants to choose a place to put a PG, it consults the rules for the pool in question; let's say rbd for rbd images (by default)
[21:25] <cbm64-1> you're right.. very useful data
[21:27] * ctrl (~ctrl@95-24-247-139.broadband.corbina.ru) Quit (Ping timeout: 480 seconds)
[21:28] * ctrl (~ctrl@128-72-251-189.broadband.corbina.ru) has joined #ceph
[21:28] <cbm64-1> dmick I would use ceph with openstack (swift) and have a lot of tutorial handy... could you give me some suggested read?
[21:30] * avoine (~avoine@modemcable121.95-163-184.mc.videotron.ca) Quit (Ping timeout: 480 seconds)
[21:30] * sstan (~chatzilla@dmzgw2.cbnco.com) Quit (Remote host closed the connection)
[21:31] <dmick> almost all of our authoritative goodness is buried in http://ceph.com/docs/
[21:32] <cbm64-1> dmick all bad vauels decreasing ... HEALTH_WARN 22 pgs backfill; 6 pgs backfilling; 14 pgs recovering; 36 pgs recovery_wait; 78 pgs stuck unclean; recovery 2398/21882 degraded (10.959%)
[21:32] <dmick> http://ceph.com/docs/master/rbd/rbd-openstack/ is what we have so far for openstack. Note that openstack doesn't imply swift
[21:33] <cbm64-1> ok
[21:34] * scuttlemonkey (~scuttlemo@217.39.7.254) Quit (Quit: This computer has gone to sleep)
[21:36] <cbm64-1> dmick: bad values still decreasing...
[21:37] * xdeller (~xdeller@broadband-77-37-224-84.nationalcablenetworks.ru) has joined #ceph
[21:39] <dmick> <thumbs up>
[21:47] <cbm64-1> dmick: tadaaa! HEALTH_OK
[21:48] <cbm64-1> dmick thank you very, very much without your helpful suggestions who knows when I'll fix my cluster...!
[21:49] <dmick> cbm64-1: good deal. Learning CRUSH placement is something that takes a while.
[21:49] <dmick> and I'm no expert yet
[21:50] <cbm64-1> yes but i suppose it's worth the effort...
[21:51] <cbm64-1> thx again and see u on #ceph !
[21:53] * cbm64-1 (c1cd3c85@ircip1.mibbit.com) has left #ceph
[21:53] * ctrl (~ctrl@128-72-251-189.broadband.corbina.ru) Quit (Ping timeout: 480 seconds)
[21:53] * bletch (c1cd3c85@ircip1.mibbit.com) Quit (Quit: http://www.mibbit.com ajax IRC Client)
[21:53] * ctrl (~ctrl@128-72-97-150.broadband.corbina.ru) has joined #ceph
[22:03] * miroslav1 (~miroslav@173-228-38-131.dsl.dynamic.sonic.net) has joined #ceph
[22:07] * rturk-away is now known as rturk
[22:09] * miroslav (~miroslav@173-228-38-131.dsl.dynamic.sonic.net) Quit (Ping timeout: 480 seconds)
[22:09] * sagelap (~sage@diaman3.lnk.telstra.net) Quit (Quit: Leaving.)
[22:09] * sagelap (~sage@diaman3.lnk.telstra.net) has joined #ceph
[22:13] * sleinen (~Adium@2001:620:0:26:d911:f3c9:6bf3:784b) Quit (Quit: Leaving.)
[22:13] * sagelap (~sage@diaman3.lnk.telstra.net) Quit (Read error: Connection reset by peer)
[22:13] * sleinen (~Adium@217-162-132-182.dynamic.hispeed.ch) has joined #ceph
[22:19] * ctrl (~ctrl@128-72-97-150.broadband.corbina.ru) Quit (Ping timeout: 480 seconds)
[22:20] * ctrl (~ctrl@128-72-97-150.broadband.corbina.ru) has joined #ceph
[22:21] * sleinen (~Adium@217-162-132-182.dynamic.hispeed.ch) Quit (Ping timeout: 480 seconds)
[22:29] * ctrl (~ctrl@128-72-97-150.broadband.corbina.ru) Quit ()
[22:31] * sagelap (~sage@diaman3.lnk.telstra.net) has joined #ceph
[22:31] <sagelap> did anyone look at wip-pool-delete?
[22:33] * sleinen (~Adium@217-162-132-182.dynamic.hispeed.ch) has joined #ceph
[22:35] * sleinen1 (~Adium@2001:620:0:26:b9c8:8cdf:90a3:6053) has joined #ceph
[22:37] * sleinen (~Adium@217-162-132-182.dynamic.hispeed.ch) Quit (Read error: Operation timed out)
[22:44] * tchmnkyz (~jeremy@0001638b.user.oftc.net) has joined #ceph
[22:45] <tchmnkyz> hey guys. I setup a new ceph cluster and i am seeing a patter where it looks like it is not balancing the load across my mds servers. It seems the first server is taking everything. Is there a config option i can set that will allow the requests to flow to all of the mds servers or something?
[22:50] <amichel> Not sure if anyone saw my question earlier about filesystem choice on rbd volumes. Is there a preferred filesystem or preferred tuning parameters for best performance/stability/etc?
[22:50] <iggy> tchmnkyz: MDSes are per pool (or maybe PG) so if you have a small number of those that seems normal
[22:50] * nhorman (~nhorman@hmsreliant.think-freely.org) Quit (Quit: Leaving)
[22:51] <tchmnkyz> i see
[22:51] <tchmnkyz> right now the way i had it setup was i have 3 "head" nodes that do just mon/map
[22:51] <iggy> tchmnkyz: but double check that with sagelap or somebody else
[22:51] <tchmnkyz> and then i have 5 OSD nodes
[22:52] * rturk is now known as rturk-away
[22:52] <iggy> amichel: if you partition the rbd device, make the partitions align to the rbd "chunk" size (4M by default I think)
[22:55] <amichel> Should I set the filesystem parameters to anything specific to address that as well? Block size or stride or stripe width or anything like that? Or is aligning the partition at 4M generally sufficient?
[23:00] <amichel> Also, if I use it as a pv for LVM, is just throwing the dataalignment parameter at it enough or do you recommend making an aligned partition and then making that partition the pv?
[23:02] <amichel> Sorry if this whole line of questioning sounds retentive, it's just one of those things it seems like it takes about the same amount of time to do right or wrong, so worth asking. I know for our traditional SAN storage, the benefits of setting these sorts of things properly can be significant as utilization goes up.
[23:06] <iggy> I'm not sure much beyond aligning partitions will make a huge difference
[23:06] <iggy> but you'd really have to double check by testing
[23:06] * sagelap (~sage@diaman3.lnk.telstra.net) Quit (Ping timeout: 480 seconds)
[23:06] <amichel> That's fair. I figured someone in here may have done that work for me :D
[23:07] <iggy> I don't know that anyone has progressed to the optimization phase
[23:07] <iggy> most people still seem to be in the getting it up and keeping it up phase
[23:08] * sagelap (~sage@182.255.120.124) has joined #ceph
[23:10] <amichel> I'll try a couple things and run some bonnie/iozones. Is there a good spot to post that kind of thing, wiki or somesuch? In case some other detail freak is looking? :)
[23:11] <nhm> amichel: I'll be doing some RBD tests shortly, so if you have results, feel free to email me. mark.nelson@inktank.com
[23:12] <nhm> amichel: I just got my 10G bonded network setup for the client, and am doing rados bench tests now. I can do about 2GB/s for large sequential reads/writes with rados bench, so it should be interesting to see how VMs do.
[23:12] <amichel> Can do.
[23:14] <amichel> I'm mounting the rbd inside a vmware vm to a cluster behind a 1G link, sadly, so I don't expect I'll hit anything that big. I might provision another host off the 10G switch the osd server is hanging off for funsies though.
[23:14] <darkfaded> nhm: 2GB/s already? woohoo
[23:16] <nhm> Yep, to localhost I was able to do around 2.2GB/s. That was 24 disks and 8 SSDs for journals.
[23:16] <nhm> tuning might get a us a little higher.
[23:17] <iggy> amichel: I'm sure there are others on the mailing list that would like to see that kind of info as well (myself included)
[23:18] <mikedawson> nhm: will you be looking at small random write IOPS in your RBD tests or are you sticking to max throughput?
[23:19] <amichel> Iggy: I'm not on any of the ceph mailing lists, which list do you mean?
[23:19] * partner (joonas@ajaton.net) has joined #ceph
[23:19] * noob2 (~noob2@ext.cscinfo.com) Quit (Quit: Leaving.)
[23:20] <dmick> ceph-devel is the list
[23:20] <dmick> ceph-devel@vger.kernel.org. Worth being on.
[23:22] <nhm> mikedawson: Yes, definitely.
[23:23] <mikedawson> nhm: good stuff! thx
[23:23] <nhm> mikedawson: it'll be interesting to see where it goes. I haven't done much with RBD yet. I've been so focused on just getting RADOS tested.
[23:31] * mikedawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[23:31] * leseb (~leseb@5ED17881.cm-7-2b.dynamic.ziggo.nl) Quit (Remote host closed the connection)
[23:34] * vata (~vata@2607:fad8:4:6:452d:e2d2:9d03:69b9) Quit (Quit: Leaving.)
[23:39] * aliguori (~anthony@cpe-70-112-157-151.austin.res.rr.com) Quit (Quit: Ex-Chat)
[23:39] * aliguori (~anthony@cpe-70-112-157-151.austin.res.rr.com) has joined #ceph
[23:39] * BillK (~BillK@124-168-240-221.dyn.iinet.net.au) has joined #ceph
[23:49] * drokita (~drokita@199.255.228.128) Quit (Quit: Leaving.)
[23:51] * wschulze1 (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) has joined #ceph
[23:51] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) Quit (Read error: Connection reset by peer)
[23:54] * leseb (~leseb@5ED17881.cm-7-2b.dynamic.ziggo.nl) has joined #ceph
[23:56] * leseb (~leseb@5ED17881.cm-7-2b.dynamic.ziggo.nl) Quit (Remote host closed the connection)
[23:57] * sleinen1 (~Adium@2001:620:0:26:b9c8:8cdf:90a3:6053) Quit (Quit: Leaving.)
[23:57] * amichel (~amichel@salty.uits.arizona.edu) Quit ()

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.