#ceph IRC Log

Index

IRC Log for 2013-03-27

Timestamps are in GMT/BST.

[0:02] * xiaoxi (~xiaoxiche@jfdmzpr04-ext.jf.intel.com) Quit (Remote host closed the connection)
[0:03] * rzerres (~ralf.zerr@static-87-79-239-211.netcologne.de) has left #ceph
[0:08] * loicd (~loic@c-76-119-91-36.hsd1.ma.comcast.net) has joined #ceph
[0:22] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[0:29] * jtang1 (~jtang@79.97.135.214) Quit (Quit: Leaving.)
[0:38] * nwat (~Adium@eduroam-233-33.ucsc.edu) Quit (Quit: Leaving.)
[0:53] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) Quit (Quit: Leaving.)
[0:55] * Cube (~Cube@12.248.40.138) Quit (Quit: Leaving.)
[0:58] * diegows (~diegows@200-081-038-029.wireless.movistar.net.ar) Quit (Ping timeout: 480 seconds)
[1:24] * xiaoxi (~xiaoxiche@134.134.139.70) has joined #ceph
[1:24] * davidz (~Adium@ip68-96-75-123.oc.oc.cox.net) Quit (Quit: Leaving.)
[1:24] * davidz (~Adium@ip68-96-75-123.oc.oc.cox.net) has joined #ceph
[1:25] * jlogan (~Thunderbi@2600:c00:3010:1:3914:d1f5:fab4:e336) Quit (Ping timeout: 480 seconds)
[1:27] * Rocky (~r.nap@188.205.52.204) has joined #ceph
[1:28] * jlogan (~Thunderbi@72.5.59.176) has joined #ceph
[1:29] * alram (~alram@38.122.20.226) Quit (Quit: leaving)
[1:30] * Cube (~Cube@cpe-76-172-67-97.socal.res.rr.com) has joined #ceph
[1:32] * Cube1 (~Cube@cpe-76-172-67-97.socal.res.rr.com) has joined #ceph
[1:32] * Cube (~Cube@cpe-76-172-67-97.socal.res.rr.com) Quit (Read error: Connection reset by peer)
[1:39] * BManojlovic (~steki@fo-d-130.180.254.37.targo.rs) Quit (Remote host closed the connection)
[1:44] * tnt (~tnt@109.130.89.104) Quit (Ping timeout: 480 seconds)
[1:49] * davidz (~Adium@ip68-96-75-123.oc.oc.cox.net) Quit (Quit: Leaving.)
[1:49] * davidz (~Adium@ip68-96-75-123.oc.oc.cox.net) has joined #ceph
[1:54] * nwat (~Adium@c-50-131-197-174.hsd1.ca.comcast.net) has joined #ceph
[1:54] <xiaoxi> joshd: is there any plan for you to fix the "nova rely on ceph.conf for monitor address" ?
[1:55] <xiaoxi> we do need to setup several different ceph cluster but using by the same openstack cluster
[1:57] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[1:57] * nwat (~Adium@c-50-131-197-174.hsd1.ca.comcast.net) has left #ceph
[1:58] * Rocky (~r.nap@188.205.52.204) Quit (Quit: Lost terminal)
[2:00] <joshd> xiaoxi: I'd like to fix it for the H release, though I don't have exact plans yet
[2:05] * esammy (~esamuels@host-2-103-103-175.as13285.net) Quit (Quit: esammy)
[2:10] <xiaoxi> joshd:ok. so we have always use multy-pool to walk around this
[2:20] * esammy (~esamuels@host-2-103-103-175.as13285.net) has joined #ceph
[2:30] * noahmehl (~noahmehl@cpe-75-186-45-161.cinci.res.rr.com) Quit (Ping timeout: 480 seconds)
[2:31] * noahmehl (~noahmehl@cpe-75-186-45-161.cinci.res.rr.com) has joined #ceph
[2:31] * buck (~buck@bender.soe.ucsc.edu) has left #ceph
[2:38] * mikedawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) has joined #ceph
[2:44] * LeaChim (~LeaChim@b0fae63d.bb.sky.com) Quit (Ping timeout: 480 seconds)
[2:47] * esammy (~esamuels@host-2-103-103-175.as13285.net) Quit (Quit: esammy)
[3:05] * chutzpah (~chutz@199.21.234.7) Quit (Quit: Leaving)
[3:08] * lightspeed (~lightspee@fw-carp-wan.ext.lspeed.org) Quit (Ping timeout: 480 seconds)
[3:23] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[3:24] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[3:43] * lightspeed (~lightspee@fw-carp-wan.ext.lspeed.org) has joined #ceph
[3:47] * jlogan (~Thunderbi@72.5.59.176) Quit (Ping timeout: 480 seconds)
[4:02] * noahmehl_ (~noahmehl@cpe-75-186-45-161.cinci.res.rr.com) has joined #ceph
[4:07] * noahmehl (~noahmehl@cpe-75-186-45-161.cinci.res.rr.com) Quit (Ping timeout: 480 seconds)
[4:07] * noahmehl_ is now known as noahmehl
[4:08] * slang1 (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) Quit (Ping timeout: 480 seconds)
[4:14] <nz_monkey_> joshd: Hi Josh, is the fix for the RBD Writeback latency fix on track for 0.61 ? I only ask as this is impacting us.
[4:14] * mikedawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) Quit (Ping timeout: 480 seconds)
[4:22] * wschulze (~wschulze@cpe-69-203-80-81.nyc.res.rr.com) Quit (Quit: Leaving.)
[5:02] * xiaoxi (~xiaoxiche@134.134.139.70) Quit (Ping timeout: 480 seconds)
[5:07] * xiaoxi (~xiaoxiche@jfdmzpr01-ext.jf.intel.com) has joined #ceph
[5:17] * loicd (~loic@c-76-119-91-36.hsd1.ma.comcast.net) Quit (Quit: Leaving.)
[5:18] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[5:20] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[5:28] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) has joined #ceph
[5:30] * drokita (~drokita@24-107-180-86.dhcp.stls.mo.charter.com) has joined #ceph
[5:30] * drokita (~drokita@24-107-180-86.dhcp.stls.mo.charter.com) Quit ()
[5:53] * sleinen (~Adium@217-162-132-182.dynamic.hispeed.ch) has joined #ceph
[5:55] * sleinen1 (~Adium@2001:620:0:25:a0e5:ffd7:a1d1:4084) has joined #ceph
[6:01] * sleinen (~Adium@217-162-132-182.dynamic.hispeed.ch) Quit (Ping timeout: 480 seconds)
[6:30] * Cube1 (~Cube@cpe-76-172-67-97.socal.res.rr.com) Quit (Quit: Leaving.)
[6:31] * Cube (~Cube@cpe-76-172-67-97.socal.res.rr.com) has joined #ceph
[6:32] * bithin (~bithin@115.249.1.61) has joined #ceph
[7:37] * sivanov (~sivanov@gw2.maxtelecom.bg) has joined #ceph
[7:37] * sivanov is now known as maxt
[7:45] <maxt> Hello all
[7:45] <maxt> i have strange problem with adding new osd-s
[7:45] <maxt> my configuration is:
[7:46] <maxt> 2 servers
[7:46] <maxt> each server with two osd-s
[7:46] <maxt> when a try to add new osd on first server everything is OK
[7:46] * sleinen1 (~Adium@2001:620:0:25:a0e5:ffd7:a1d1:4084) Quit (Quit: Leaving.)
[7:46] * sleinen (~Adium@217-162-132-182.dynamic.hispeed.ch) has joined #ceph
[7:47] * Rocky (~r.nap@188.205.52.204) has joined #ceph
[7:47] <maxt> when i try to add new osd on second server - i do not see osd
[7:48] <maxt> there is crush map
[7:48] <maxt> sorry i mean map tree
[7:48] <maxt> -1 100 host alpha
[7:49] <maxt> 110 2 osd.110 up 1
[7:49] <maxt> 111 2 osd.111 up 1
[7:49] <maxt> -2 100 host beta
[7:49] <maxt> 120 2 osd.120 up 1
[7:49] <maxt> 121 2 osd.121 up 1
[7:51] <maxt> It seems that the second server not listening on cluster address
[7:55] * sleinen (~Adium@217-162-132-182.dynamic.hispeed.ch) Quit (Ping timeout: 480 seconds)
[7:58] * esammy (~esamuels@host-2-103-103-175.as13285.net) has joined #ceph
[7:59] * norbi (~nonline@buerogw01.ispgateway.de) has joined #ceph
[8:00] <norbi> mornin #ceph
[8:01] * Kioob (~kioob@2a01:e35:2432:58a0:21e:8cff:fe07:45b6) Quit (Quit: Leaving.)
[8:23] * tnt (~tnt@109.130.89.104) has joined #ceph
[8:24] * terje1 (~Adium@c-67-176-69-220.hsd1.co.comcast.net) has joined #ceph
[8:26] * mcclurmc_laptop (~mcclurmc@cpc10-cmbg15-2-0-cust205.5-4.cable.virginmedia.com) has joined #ceph
[8:27] * terje (~Adium@c-67-176-69-220.hsd1.co.comcast.net) Quit (Ping timeout: 480 seconds)
[8:55] * sleinen (~Adium@2001:620:0:25:e5f6:ef94:d461:9cb4) has joined #ceph
[8:57] * jtang1 (~jtang@79.97.135.214) has joined #ceph
[9:01] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) has joined #ceph
[9:06] <baz_> hi, can anyone help with a failed assert on an osd?
[9:06] * Vjarjadian (~IceChat77@5ad6d005.bb.sky.com) Quit (Quit: Easy as 3.14159265358979323846... )
[9:07] <xiaoxi> what assert
[9:07] <baz_> @xiaoxi: "join on thread that was never started"
[9:08] <baz_> seems to be the same as issue http://tracker.ceph.com/issues/4357
[9:09] <baz_> working with version 0.56.4 on Ubuntu 12.04
[9:09] * hybrid512 (~walid@LPoitiers-156-86-25-85.w193-248.abo.wanadoo.fr) has joined #ceph
[9:10] * gerard_dethier (~Thunderbi@85.234.217.115.static.edpnet.net) has joined #ceph
[9:11] <xiaoxi> sorry :( cannot provide any help
[9:15] <baz_> @xiaoxi: thanks anyway :)
[9:19] * tnt (~tnt@109.130.89.104) Quit (Ping timeout: 480 seconds)
[9:21] * eschnou (~eschnou@85.234.217.115.static.edpnet.net) has joined #ceph
[9:26] * Cube (~Cube@cpe-76-172-67-97.socal.res.rr.com) Quit (Quit: Leaving.)
[9:27] * BManojlovic (~steki@91.195.39.5) has joined #ceph
[9:30] * tnt (~tnt@212-166-48-236.win.be) has joined #ceph
[9:36] * jtang1 (~jtang@79.97.135.214) Quit (Quit: Leaving.)
[9:39] * maxt (~sivanov@gw2.maxtelecom.bg) Quit (Ping timeout: 480 seconds)
[9:47] * ScOut3R (~ScOut3R@212.96.47.215) has joined #ceph
[9:50] * l0nk (~alex@83.167.43.235) has joined #ceph
[9:53] * leseb (~Adium@83.167.43.235) has joined #ceph
[9:54] * topro (~topro@host-62-245-142-50.customer.m-online.net) Quit (Quit: Konversation terminated!)
[9:59] * topro (~topro@host-62-245-142-50.customer.m-online.net) has joined #ceph
[9:59] * askort (~askort@2a02:6b8:0:401:9f1:9da4:4351:82bc) has joined #ceph
[9:59] * tryggvil (~tryggvil@17-80-126-149.ftth.simafelagid.is) Quit (Quit: tryggvil)
[10:00] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[10:00] * xiaoxi (~xiaoxiche@jfdmzpr01-ext.jf.intel.com) Quit (Ping timeout: 480 seconds)
[10:02] * LeaChim (~LeaChim@b0fae63d.bb.sky.com) has joined #ceph
[10:11] * dosaboy (~gizmo@faun.canonical.com) has joined #ceph
[10:14] * maxt (~sivanov@wifi-gw.3dc.net) has joined #ceph
[10:15] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[10:16] * jjgalvez1 (~jjgalvez@cpe-76-175-30-67.socal.res.rr.com) has joined #ceph
[10:17] * jjgalvez (~jjgalvez@cpe-76-175-30-67.socal.res.rr.com) Quit (Ping timeout: 480 seconds)
[10:20] * stacker666 (~stacker66@206.pool85-61-191.dynamic.orange.es) has joined #ceph
[10:22] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[10:22] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[10:25] * jtang1 (~jtang@2001:770:10:500:a53a:3a80:3afa:9b1c) has joined #ceph
[10:31] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) Quit (Remote host closed the connection)
[10:37] * jjgalvez (~jjgalvez@cpe-76-175-30-67.socal.res.rr.com) has joined #ceph
[10:38] * maxiz (~pfliu@222.128.152.208) Quit (Ping timeout: 480 seconds)
[10:38] * maxt (~sivanov@wifi-gw.3dc.net) Quit (Ping timeout: 480 seconds)
[10:39] * Rocky (~r.nap@188.205.52.204) Quit (Quit: Lost terminal)
[10:42] * jjgalvez1 (~jjgalvez@cpe-76-175-30-67.socal.res.rr.com) Quit (Ping timeout: 480 seconds)
[10:45] * jtang2 (~jtang@2001:770:10:500:3d22:b2b:cf17:51a1) has joined #ceph
[10:48] * maxiz (~pfliu@222.128.145.154) has joined #ceph
[10:51] * jtang1 (~jtang@2001:770:10:500:a53a:3a80:3afa:9b1c) Quit (Ping timeout: 480 seconds)
[11:02] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) has joined #ceph
[11:08] * jjgalvez (~jjgalvez@cpe-76-175-30-67.socal.res.rr.com) Quit (Quit: Leaving.)
[11:09] * maxt (~sivanov@gw2.maxtelecom.bg) has joined #ceph
[11:18] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) Quit (Read error: Operation timed out)
[11:25] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) Quit (Ping timeout: 480 seconds)
[11:36] * loicd (~loic@c-76-119-91-36.hsd1.ma.comcast.net) has joined #ceph
[11:38] * Rocky (~r.nap@188.205.52.204) has joined #ceph
[11:39] * maxiz (~pfliu@222.128.145.154) Quit (Ping timeout: 480 seconds)
[11:47] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) has joined #ceph
[11:51] * diegows (~diegows@190.190.2.126) has joined #ceph
[11:51] * rzerres (~ralf.zerr@bermuda.daywalker-studios.de) has joined #ceph
[11:51] * maxiz (~pfliu@222.128.145.154) has joined #ceph
[11:58] * norbi (~nonline@buerogw01.ispgateway.de) Quit (Quit: Miranda IM! Smaller, Faster, Easier. http://miranda-im.org)
[12:05] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) has joined #ceph
[12:06] * mikedawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) has joined #ceph
[12:09] * sleinen (~Adium@2001:620:0:25:e5f6:ef94:d461:9cb4) Quit (Quit: Leaving.)
[12:09] * sleinen (~Adium@130.59.94.70) has joined #ceph
[12:11] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) Quit (Ping timeout: 480 seconds)
[12:15] * sleinen1 (~Adium@2001:620:0:25:c962:dd7:7c6b:245a) has joined #ceph
[12:15] * mikedawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) Quit (Ping timeout: 480 seconds)
[12:21] * sleinen (~Adium@130.59.94.70) Quit (Ping timeout: 480 seconds)
[12:28] * maxiz (~pfliu@222.128.145.154) Quit (Ping timeout: 480 seconds)
[12:34] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) has joined #ceph
[12:35] * agh (~agh@www.nowhere-else.org) has joined #ceph
[12:36] <agh> Hello, I do not understand one thing...
[12:36] * leseb (~Adium@83.167.43.235) Quit (Quit: Leaving.)
[12:36] <agh> I have a CEPH cluster with 4 nodes, with each 8 disks of 3 TB.
[12:37] <agh> So, in total, I have 8*3*4 TB = 96 TB of raw storage
[12:37] <agh> well.
[12:37] <agh> now, i have a replica size of 2 on every pools... so normally, I should have a useable space of 96/2 = 48 TB
[12:38] <agh> ... Or, when i do a "ceph status" i have :
[12:38] <agh> pgmap v2567: 8336 pgs: 8336 active+clean; 0 bytes data, 1562 MB used, 89379 GB / 89380 GB avail
[12:38] <agh> so, 89380 GB = 87 TB
[12:38] <agh> ... Why ? Where is the issue ?
[12:39] <agh> Is there a way to be ABSOLUTLY sure that each object is replicated twice ?
[12:41] <andreask> looks like GiB vs GB and TiB vs TB issue
[12:42] <darkfader> gb vs GB makes that more visible
[12:43] <darkfader> but yeah, plus agh if it tells you 87TB available that means, regardless of replication policy
[12:43] <darkfader> so storing data with 2 replicas will double the space "data" uses
[12:44] <agh> mmm. i don't understand the ouput of ceph status
[12:44] <agh> let's imagine i have a 100 GB raw storage cluster
[12:44] <agh> pool with replica size=2
[12:44] <agh> what will the output of ceph status ?
[12:45] <darkfader> at start it'd say almost 100GB free. then if you store 1GB it'll say almost 98GB free
[12:45] <agh> 0 bytes data, 1000 MB used, 99 GB / 100 GB avail
[12:45] <darkfader> and if you store 10GB it'll be 80GB free
[12:45] <agh> ?
[12:45] <agh> ok, so the output does not take care of the replication level...
[12:46] <darkfader> it tells you how much is free
[12:46] <darkfader> not free if you used that certain replication level on all stuff you store in there
[12:46] <darkfader> because ther could be different levels etc.
[12:46] <agh> mm. understand
[12:47] <darkfader> great :)
[12:47] <agh> But is there a way to get the total useable free space ?
[12:47] <agh> (last question, after i go to lunch :) )
[12:49] <darkfader> i don't think there's a command like it, maybe i'm just not thinking the right way now
[12:49] <darkfader> it'll turn out no problem i promise
[12:49] <darkfader> oh at least it was never one for me
[12:50] <darkfader> agh: what kind of stuff do you want to store?
[12:50] <darkfader> vm images? object-object-object-ish stuff?
[12:55] <rzerres> @joao: hey there. Again i would be please to get your insite view about ceph - mon. I thought everything is fine, but it seems i'm wrong
[12:55] <joao> rzerres, what seems to be wrong?
[12:55] <rzerres> cluster is working, ceph -s, ceph -w do what they should do.
[12:56] <rzerres> actual output of "ceph mon stat': e4: 3 mons at {0=172.16.0.3:6789/0,1=172.16.0.4:6789/0,2=172.16.0.2:6789/0}, election epoch 290, quorum 0,1 0,2
[12:57] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) Quit (Ping timeout: 480 seconds)
[12:57] <rzerres> here my question: is mon.0 talking to mon.1 and mon.0 talking to mon.2 with quorum, but mon.1 not in quorum with mon.2?
[12:59] <rzerres> when i do reboot one phyical cluster-server (running mon, mds, osd) the monitor should still react, since there is a majority mon's for the cluster, right?
[13:00] <joao> rzerres, on 'quorum 0,1 0,2', what that means is that your quorum is composed on the monitors with rank 0 and 1 (from 'quorum 0,1'), whose names are 0 and 2
[13:00] <joao> so it tells you that mon.0 and mon.2 are in quorum, with mon.0 having rank 0 (i.e., the leader), and mon.2 having rank 1 (i.e., a peon)
[13:00] <joao> mon.1 is not part of the quorum
[13:02] <joao> and now that I think of it, that's probably why we use letters as the monitor's ids
[13:02] <joao> because that representation is rather confusing
[13:03] <rzerres> so for clarificasion i should rename to mon.a, mon.b and mon.c in my case inside the ceph.conf and restart the monitors? will the filesystems overcome that?
[13:06] <joao> you don't have to rename anything
[13:06] <joao> renaming monitors seldom solves whatever may be the problem
[13:06] * leseb (~Adium@83.167.43.235) has joined #ceph
[13:07] <joao> what it looks like is that you expected to have all the 3 monitors in the quorum but only two of them actually are
[13:07] <rzerres> well, thats clear, but i get more in line with usual behaviour
[13:07] <joao> so the question is "why isn't mon.1 part of the quorum"
[13:07] <rzerres> that's what i have seen. yes
[13:07] <joao> and only by looking at the logs would we have a better picture of what's happening
[13:09] <rzerres> ok. so i should activate debugging for mon = 20 and paxos = 20 for mon.1 and pastebin?
[13:10] <joao> and debug ms = 1
[13:10] <joao> yes, please
[13:13] <rzerres> bye the way, can i use "ceph mon tell 1 -- injectargs 'debug-ms=1 debug-paxos=20 debug-mon=20'" to intercept the monitor dynamicly?
[13:13] <joao> you could if the monitor was in the quorum
[13:13] <joao> I think it is possible to do it via the admin socket, but can't recall if that's so
[13:16] <joao> rzerres, I don't think you can inject args via the admin socket; but you can however check how's them monitor faring
[13:16] <joao> 'ceph --admin-daemons /path/to/mon.1.asok mon_status'
[13:17] * leseb (~Adium@83.167.43.235) Quit (Ping timeout: 480 seconds)
[13:19] <rzerres> ok. got it
[13:19] <rzerres> now i restarted the mon.1 and output of ceph mon stat:
[13:19] <rzerres> e4: 3 mons at {0=172.16.0.3:6789/0,1=172.16.0.4:6789/0,2=172.16.0.2:6789/0}, election epoch 292, quorum 0,1,2 0,1,2
[13:19] <rzerres> now they are all in quorum.
[13:19] <rzerres> strange
[13:20] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) has joined #ceph
[13:20] <rzerres> log did show: mon.1@2(electing).elector(284) init, last seen epoch 284
[13:22] <joao> well, monitors go through an election to form a quorum
[13:23] <joao> that's how they assess who's the leader and who's the peons
[13:24] <rzerres> so, all went down as expected. Wonder why the election didn't got through yesterday when i started the cluster-node ...
[13:25] * maxiz (~pfliu@222.128.145.154) has joined #ceph
[13:25] <rzerres> again my question, can i rename the mon sections in ceph.conf for the defined monitors, or do i have to build up new mkfs when renaming stuff in v0.59+
[13:26] * wschulze (~wschulze@cpe-69-203-80-81.nyc.res.rr.com) has joined #ceph
[13:30] <rzerres> @joao: got it with ceph --admin-daemon <path_to_mon_asok_file> mon_status
[13:30] <askort> Hello! Can you explain how i can set limit for number of ceph-osd threads?
[13:30] <askort> I set "osd [op|disk|recovery] threads = 1", but there are running around 160 threads per one ceph-osd process.
[13:31] <joao> rzerres, you really can't just rename the monitors on ceph.conf
[13:31] <joao> you'd have to go through with removing mon.0 and add mon.a; then remove mon.1 and add mon.b; then remove mon.2 and add mon.c
[13:32] <joao> rzerres, basically following this: http://ceph.com/docs/master/rados/operations/add-or-rm-mons/#changing-a-monitor-s-ip-address-the-right-way
[13:32] <joao> but the focus being the names instead of the ips
[13:33] * agh (~agh@www.nowhere-else.org) Quit (Remote host closed the connection)
[13:33] <joao> but given you're probably going to leave them in the same host, I have a strong feeling towards having to remove the monitor prior to adding it to the monmap, with a new name, again
[13:34] <joao> and you'll have to adjust your ceph.conf in order for your daemons to find the monitors
[13:35] <rzerres> joao: will take this a practice example. yes, i will remove the working mon first, then add the new one ...
[13:35] <joao> rzerres, if you don't manage to always have 2 monitors up, out of 3, your quorum will break and you will be unable to add the new monitor
[13:36] <joao> I mean, if you have 3 monitors and remove 1 of them in order to readd it with a different name, you will have to make sure you keep the other 2 going
[13:39] * slang (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) has joined #ceph
[13:42] * leseb (~Adium@3.46-14-84.ripe.coltfrance.com) has joined #ceph
[13:43] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) Quit (Ping timeout: 480 seconds)
[13:46] * Romeo (~romeo@198.144.195.85) Quit (Read error: Connection reset by peer)
[13:46] * Romeo (~romeo@198.144.195.85) has joined #ceph
[13:52] * nhorman (~nhorman@2001:470:8:a08:7aac:c0ff:fec2:933b) has joined #ceph
[13:52] * mcclurmc_laptop (~mcclurmc@cpc10-cmbg15-2-0-cust205.5-4.cable.virginmedia.com) Quit (Ping timeout: 480 seconds)
[14:01] * loicd (~loic@c-76-119-91-36.hsd1.ma.comcast.net) Quit (Quit: Leaving.)
[14:04] <askort> ?
[14:04] * davidz (~Adium@ip68-96-75-123.oc.oc.cox.net) Quit (Quit: Leaving.)
[14:05] * mikedawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) has joined #ceph
[14:06] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) has joined #ceph
[14:08] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) Quit (Remote host closed the connection)
[14:08] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) has joined #ceph
[14:09] * ScOut3R_ (~ScOut3R@212.96.47.215) has joined #ceph
[14:09] <Azrael> anybody around who can help with ceph + chef?
[14:11] * sleinen (~Adium@130.59.94.70) has joined #ceph
[14:16] * ScOut3R (~ScOut3R@212.96.47.215) Quit (Ping timeout: 480 seconds)
[14:19] * sleinen1 (~Adium@2001:620:0:25:c962:dd7:7c6b:245a) Quit (Ping timeout: 480 seconds)
[14:19] * sleinen (~Adium@130.59.94.70) Quit (Ping timeout: 480 seconds)
[14:20] * sleinen (~Adium@130.59.94.70) has joined #ceph
[14:23] * sleinen1 (~Adium@2001:620:0:26:c59d:8069:fe3c:7f9b) has joined #ceph
[14:25] * sleinen1 (~Adium@2001:620:0:26:c59d:8069:fe3c:7f9b) Quit ()
[14:25] * sleinen (~Adium@130.59.94.70) Quit (Read error: Connection reset by peer)
[14:28] <askort> It seems like "osd op threads = 0" doesn't disable multi-threading, but disable all osd operations.
[14:29] * drokita (~drokita@199.255.228.128) has joined #ceph
[14:32] * timmclaughlin (~timmclaug@69.170.148.179) has joined #ceph
[14:38] * portante|afk is now known as portante
[14:39] <matt_> askort, are you referring to the threads just accessing the disk or total threads of the ceph-osd process?
[14:41] <askort> i think all threads
[14:42] <matt_> ceph uses a different thread for just about everything and since you're just altering the disk threads I don't think it will have much impact
[14:42] <matt_> ... I think anyway
[14:43] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) Quit (Remote host closed the connection)
[14:44] <askort> Changing osd_op_threads from 1 to 0, decrease number of threads from ~150 to ~40.
[14:45] <matt_> Interesting, what the difference between 1 and 2?
[14:45] <askort> I have 10 osd per server and that provides over 1.5k threads
[14:45] <elder> 1
[14:45] <tchmnkyz> hey guys, i just had a issue where 3 nodes of 5 in my cluster just dropped (the OSD) was a memory issue. The physical servers stayed running but the OSD daemon died
[14:46] <tchmnkyz> when i check the syslog i see quite a few entries pointing to the OSD eating the entire memory on the server, what i dont get is that i have 48GB of ram in each osd.
[14:47] <tchmnkyz> there is no reason that i could see why it would eat 48gb of ram like that
[14:49] * BManojlovic (~steki@91.195.39.5) Quit (Quit: Ja odoh a vi sta 'ocete...)
[14:49] * BManojlovic (~steki@91.195.39.5) has joined #ceph
[14:52] * timmclau_ (~timmclaug@69.170.148.179) has joined #ceph
[14:53] * sleinen (~Adium@130.59.94.70) has joined #ceph
[14:54] * sleinen (~Adium@130.59.94.70) Quit ()
[14:54] <askort> tchmnkyz, with error "ceph-osd: page allocation failure: order:2, mode:0x4020" ?
[14:56] <tchmnkyz> similar yes
[14:56] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) has joined #ceph
[14:56] <tchmnkyz> ceph-osd: page allocation failure. order:5, mode:0x40d0
[14:56] * aliguori (~anthony@cpe-70-112-157-87.austin.res.rr.com) has joined #ceph
[14:57] * PerlStalker (~PerlStalk@72.166.192.70) has joined #ceph
[14:58] <askort> same problem :(
[14:58] <tchmnkyz> what kernel you using?
[14:58] <tchmnkyz> i am running debian stable
[14:59] <askort> 10 ceph-osd, 128 GB RAM
[14:59] <askort> 3.4.20-030420-generic from kernel-ppa
[14:59] <tchmnkyz> deb or ub
[14:59] <askort> 12.04
[15:00] * timmclaughlin (~timmclaug@69.170.148.179) Quit (Ping timeout: 480 seconds)
[15:00] <tchmnkyz> i just dont get what would use that much memmory in a node like that
[15:02] <tchmnkyz> like now since rebooting it an hour ago it is already using 32 GB of ram
[15:03] <askort> i have references to nic driver in syslog, but i'm not 100% sure
[15:04] * loicd (~loic@c-98-229-30-137.hsd1.ma.comcast.net) has joined #ceph
[15:09] * l0nk (~alex@83.167.43.235) Quit (Quit: Leaving.)
[15:14] * xiaoxi (~xiaoxiche@134.134.139.74) has joined #ceph
[15:15] <t0rn> where should i look if when using rbd map it is making the device at /dev/rbd1 rather than /dev/rbd/{poolname}/{imagename} ?
[15:19] * mcclurmc_laptop (~mcclurmc@firewall.ctxuk.citrix.com) has joined #ceph
[15:20] * barryo (~borourke@cumberdale.ph.ed.ac.uk) Quit (Ping timeout: 480 seconds)
[15:24] <mattch> t0rn: Probably a missing udev rule
[15:29] * barryo (~borourke@cumberdale.ph.ed.ac.uk) has joined #ceph
[15:30] * jskinner (~jskinner@69.170.148.179) has joined #ceph
[15:30] * sleinen (~Adium@130.59.94.70) has joined #ceph
[15:32] * sleinen1 (~Adium@2001:620:0:25:d407:9744:2527:3060) has joined #ceph
[15:34] <tchmnkyz> askort: what nic you have in your server?
[15:34] <tchmnkyz> i am using supermicro x8 boards with dual intel nic
[15:34] <askort> Mellanox ConnectX-3
[15:34] <tchmnkyz> but using IB as the backend for ceph
[15:35] <askort> also Intel I350, but for ceph only mellanox
[15:38] <tchmnkyz> Mellanox Technologies MT25204
[15:38] <tchmnkyz> maybe a bug with ipoib
[15:38] <tchmnkyz> one commonality between us
[15:38] * sleinen (~Adium@130.59.94.70) Quit (Ping timeout: 480 seconds)
[15:43] <darkfader> yay infiniband porn
[15:46] * liiwi (liiwi@77.86.213.88) Quit (Quit: leaving)
[15:46] * liiwi (liiwi@idle.fi) has joined #ceph
[15:50] * l0nk (~alex@83.167.43.235) has joined #ceph
[15:52] * liiwi (liiwi@idle.fi) Quit (Quit: leaving)
[15:52] * timmclau_ (~timmclaug@69.170.148.179) Quit (Remote host closed the connection)
[15:53] * timmclaughlin (~timmclaug@69.170.148.179) has joined #ceph
[15:53] * liiwi (liiwi@idle.fi) has joined #ceph
[15:55] * portante is now known as portante|afk
[15:57] * loicd (~loic@c-98-229-30-137.hsd1.ma.comcast.net) Quit (Quit: Leaving.)
[15:58] * BManojlovic (~steki@91.195.39.5) Quit (Remote host closed the connection)
[15:59] * timmclau_ (~timmclaug@69.170.148.179) has joined #ceph
[16:00] <tchmnkyz> anyone else having problems with infinband and ceph?
[16:04] <rzerres> joao: finaly -> ection epoch 314, quorum 0,1,2 a,b,c
[16:06] <rzerres> joao: before i forget the correct order of the commands, i made myself some smal shell scritps to do the job. so it succeeded! :)
[16:06] * timmclaughlin (~timmclaug@69.170.148.179) Quit (Ping timeout: 480 seconds)
[16:07] * loicd (~loic@74-94-156-210-NewEngland.hfc.comcastbusiness.net) has joined #ceph
[16:07] <joao> rzerres, cool!
[16:08] <maxt> Hello all
[16:08] <maxt> I have a problem with adding new osd
[16:09] <rzerres> joao: in the end, i can get rid of all the old v1 structures in the old monitor pathes, right?
[16:10] <maxt> I have two servers(alpha, beta). When i try to add new osd in alpha server everything is OK, when i try to add osd to second server(beta) i not view osd in "ceph osd tree"
[16:10] <rzerres> joao: that means v0.59+ (mon v2) only uses <monpath>/store.db + <monpath>/keyring
[16:10] <maxt> daemon is runnig no errors
[16:10] <maxt> any ideas?
[16:11] <joao> rzerres, yes, pretty much, but we don't really call it 'old v1' anything
[16:14] * ron-slc (~Ron@173-165-129-125-utah.hfc.comcastbusiness.net) has joined #ceph
[16:15] * sleinen (~Adium@2001:620:0:2d:64ab:b7c:1175:cabc) has joined #ceph
[16:20] <scuttlemonkey_> maxt: what are you doing to add the osd?
[16:20] * scuttlemonkey_ is now known as scuttlemonkey
[16:20] <maxt> http://ceph.com/docs/master/rados/operations/add-or-rm-osds/?highlight=remove%20osd
[16:20] <maxt> step by step
[16:21] <scuttlemonkey> sure, but are you doing it by hand? with chef? with ceph-deploy?
[16:21] <maxt> with ceph-deploy
[16:22] * sleinen1 (~Adium@2001:620:0:25:d407:9744:2527:3060) Quit (Ping timeout: 480 seconds)
[16:23] * jlogan1 (~Thunderbi@2600:c00:3010:1:8527:f7:8cac:c375) has joined #ceph
[16:23] * sleinen (~Adium@2001:620:0:2d:64ab:b7c:1175:cabc) Quit (Ping timeout: 480 seconds)
[16:24] <scuttlemonkey> so from your main bootstrap box you can 'ceph-deploy osd host1:disk' and it work, but 'ceph-deploy osd host2:disk' fails?
[16:24] <scuttlemonkey> do you see any movement on ceph -w when you try host2?
[16:26] <scuttlemonkey> or wait...that may have changed with the new ceph-deploy
[16:26] <scuttlemonkey> it's create and activate now, right?
[16:26] <scuttlemonkey> https://github.com/ceph/ceph-deploy
[16:26] <maxt> yes is active
[16:28] <scuttlemonkey> sorry, there were a lot of questions there...
[16:28] <scuttlemonkey> 1) host1 works, host2 fails?
[16:28] <maxt> yes
[16:28] <scuttlemonkey> 2) if host2 is failing does ceph -w show anything when you do it (keep it open in another window)
[16:28] <maxt> no no
[16:28] <maxt> look
[16:29] <maxt> Host 1 have 4 osds, 2 when creating the cluster, 2 added after
[16:29] <maxt> Host 2 have 2 osds when creating the cluster
[16:30] <maxt> so... i think 1 and 2 works
[16:30] <scuttlemonkey> ok, so it's the last 2 disks on host2 that are bombing?
[16:31] <maxt> i just try to add third osd
[16:31] <maxt> now osd 3 daemon is running
[16:31] <maxt> but cluster not see this osd 3
[16:31] <scuttlemonkey> ok, the mons don't see it at all?
[16:32] <maxt> yes
[16:32] <scuttlemonkey> is there anything in your /var/log/ceph or /var/log/ceph/osd.3?
[16:33] <maxt> yes look this is are last two lines
[16:33] <maxt> 2013-03-27 17:07:49.912040 7f69cc822780 1 journal _open /var/ceph/osd/osd.122/journal fd 27: 10485760000 bytes, block size 4096 bytes, directio = 1, aio = 0
[16:33] <maxt> 2013-03-27 17:07:49.934791 7f69cc822780 1 journal _open /var/ceph/osd/osd.122/journal fd 27: 10485760000 bytes, block size 4096 bytes, directio = 1, aio = 0
[16:33] <maxt> and there are last two lines from osd 2
[16:33] * tziOm (~bjornar@237-25-11.connect.netcom.no) has joined #ceph
[16:34] <maxt> 2013-03-27 14:16:25.975302 7fc09d7bc780 1 journal _open /var/ceph/osd/osd.120/journal fd 30: 10485760000 bytes, block size 4096 bytes, directio = 1, aio = 0
[16:34] <maxt> 2013-03-27 14:16:25.981988 7fc09d7bc780 1 journal _open /var/ceph/osd/osd.120/journal fd 30: 10485760000 bytes, block size 4096 bytes, directio = 1, aio = 0
[16:34] <maxt> 2013-03-27 14:16:26.031521 7fc087921700 0 -- 10.241.0.11:6800/31356 >> 10.241.0.11:6803/12965 pipe(0x2f66500 sd=36 :0 s=1 pgs=0 cs=0 l=0).fault with nothing to send, going to standby
[16:34] <maxt> 2013-03-27 14:16:26.031733 7fc087d25700 0 -- 10.241.0.11:6800/31356 >> 10.241.0.10:6810/18586 pipe(0x26e5780 sd=32 :0 s=1 pgs=0 cs=0 l=0).fault with nothing to send, going to standby
[16:35] <maxt> do you want to see "ceph osd tree"?
[16:36] <scuttlemonkey> can you pastebin ceph -s first?
[16:36] <maxt> health HEALTH_WARN 21 pgs degraded; 115 pgs stuck unclean
[16:36] <maxt> monmap e3: 2 mons at {a=192.168.11.141:6789/0,b=192.168.11.130:6789/0}, election epoch 60, quorum 0,1 a,b
[16:36] <maxt> osdmap e396: 6 osds: 6 up, 6 in
[16:36] <maxt> pgmap v2318: 968 pgs: 853 active+clean, 94 active+remapped, 21 active+degraded; 3221 MB data, 66838 MB used, 11107 GB / 11172 GB avail
[16:36] <maxt> mdsmap e1: 0/0/1 up
[16:36] <scuttlemonkey> that 'fault with nothing to send' usually just means that one of the messenger's sockets hasn't seen activity for a while and hit timeout
[16:37] * stacker100 (~stacker66@152.pool85-58-180.dynamic.orange.es) has joined #ceph
[16:37] <scuttlemonkey> supposedly it will reestablish the session cleanly if either end has a message to send
[16:38] <scuttlemonkey> looks like you have something else going on though
[16:38] <scuttlemonkey> some degraded pgs
[16:38] * stacker666 (~stacker66@206.pool85-61-191.dynamic.orange.es) Quit (Read error: Operation timed out)
[16:38] <scuttlemonkey> pastebin.com
[16:38] <scuttlemonkey> can you drop your crushmap and ceph.conf in there?
[16:39] <maxt> okay just a second
[16:41] <maxt> http://pastebin.com/di2ahtYJ
[16:45] <thorus> I have a test ceph cluster, with 2 machines and installed and vm with proxmox on it, when I'm shutting down one node, the whole cluster doesn't work anymore, thats not how its supposed to be right? It should work in an degraded state
[16:45] <scuttlemonkey> maxt: so it looks like gamma is still in the crush rules, even though the osds are commented out
[16:46] <scuttlemonkey> until you spin that up I might comment out rack_i4 and the rack_i4 item in 3DC
[16:48] <maxt> All have some problem with adding osd-s in gamma server
[16:48] <maxt> All = I :)
[16:48] <maxt> Remove gamma from crush?
[16:49] <scuttlemonkey> if you want to look and see first we can just 'ceph -s pg dump | grep unclean'
[16:49] <scuttlemonkey> and see if it's try to map to gamma
[16:49] <scuttlemonkey> trying*
[16:50] <scuttlemonkey> er.. s/unclean/degraded
[16:51] <maxt> with grep degraded report only osd 110, 111
[16:53] <scuttlemonkey> can you 'ceph pg {PG#} query' one of the degraded pgs?
[16:53] <scuttlemonkey> I wanna see what the problem is
[16:53] <scuttlemonkey> you'll want to pastebin it
[16:54] <maxt> http://pastebin.com/1951uDdA
[16:55] <scuttlemonkey> yeah, so it only has 1 replica
[16:55] <scuttlemonkey> my guess is it tried to assign it to gamma, but there was no corresponding host/osd, so it just failed
[16:55] <scuttlemonkey> and is sitting in a degraded state
[16:56] <scuttlemonkey> lets rip gamma out of the crushmap and see if we can't get it to health ok so we can see what's really going on with adding osds on beta
[16:56] <maxt> yes this is true
[16:56] <maxt> okay i will remove gamma from crush
[16:57] * gerard_dethier (~Thunderbi@85.234.217.115.static.edpnet.net) Quit (Quit: gerard_dethier)
[16:58] <maxt> done
[16:58] <scuttlemonkey> give it a minute or two to rebalance
[16:58] <maxt> okay
[16:58] <scuttlemonkey> and lets see if it doesn't catch on
[17:02] * tziOm (~bjornar@237-25-11.connect.netcom.no) Quit (Ping timeout: 480 seconds)
[17:03] <maxt> catch on?
[17:03] <maxt> HEALTH_WARN 21 pgs degraded; 115 pgs stuck unclean; recovery recovering 4 o/s, 17822KB/s
[17:03] <scuttlemonkey> yeah, any movement on ceph -w?
[17:03] <maxt> this is the last
[17:03] <maxt> state
[17:03] <maxt> 2013-03-27 18:02:21.429977 mon.0 [INF] pgmap v2381: 968 pgs: 853 active+clean, 94 active+remapped, 21 active+degraded; 3221 MB data, 67129 MB used, 11107 GB / 11172 GB avail; recovering 4 o/s, 17822KB/s
[17:04] <scuttlemonkey> so it is recovering
[17:04] <maxt> HEALTH_WARN 21 pgs degraded; 115 pgs stuck unclean
[17:04] <maxt> i think is done
[17:06] <scuttlemonkey> hang on, brb
[17:06] * scuttlemonkey (~scuttlemo@c-69-244-181-5.hsd1.mi.comcast.net) Quit (Quit: my troubles seem so far away, now yours are too...)
[17:07] * scuttlemonkey (~scuttlemo@c-69-244-181-5.hsd1.mi.comcast.net) has joined #ceph
[17:07] * ChanServ sets mode +o scuttlemonkey
[17:08] * BillK (~BillK@58-7-172-123.dyn.iinet.net.au) Quit (Ping timeout: 480 seconds)
[17:08] <scuttlemonkey> maxt: can you pastebin your osd tree?
[17:08] <maxt> http://pastebin.com/XzAnxTVW
[17:10] * BManojlovic (~steki@91.195.39.5) has joined #ceph
[17:11] * tziOm (~bjornar@ip-202-1-149-91.dialup.ice.net) has joined #ceph
[17:16] <thorus> ok i should get more clearly: i have two physical maschines, on every one runs an osd and mon, on the third machines runs a proxmox, where I installed an vm with the image lying in the ceph cluster, when I now stop one mon on one node, the whole cluster stops working, stopping one mon gets me the expected result, an degraded cluster. Do I have to configure something to ceph to also work with only on mon availible?
[17:18] <scuttlemonkey> maxt: lets see if we can't make the new osd tell us what's going on with a stop ceph-osd id=122 and then ceph-osd -n 122 --debug-osd 10 --debug-ms 1
[17:19] <maxt> stop 122 and start with debug options?
[17:19] * sleinen (~Adium@2001:620:0:25:4408:2cb8:dbe8:d545) has joined #ceph
[17:19] <scuttlemonkey> yeah
[17:20] <scuttlemonkey> thorus: you need enough mons to still reach quorum
[17:20] <scuttlemonkey> we recommend using at least 3 mons
[17:20] * bithin (~bithin@115.249.1.61) Quit (Quit: Leaving)
[17:20] <thorus> scuttlemonkey: so there is no way to do it with 2 and get an degraded cluster when one fails?
[17:21] <scuttlemonkey> thorus: if you have that setup, when one mon goes down you don't have a quorum anymore so it'll block trying to reach the cluster
[17:21] * davidz (~Adium@ip68-96-75-123.oc.oc.cox.net) has joined #ceph
[17:21] <scuttlemonkey> usually I see a small (odd) number of mons
[17:22] <scuttlemonkey> 1 or 3 are the most common (1 for test, 3 for prod) but some envs I have seen have 5
[17:22] <thorus> scuttlemonkey: ok so I need to get another mon
[17:23] * buck (~buck@bender.soe.ucsc.edu) has joined #ceph
[17:24] <scuttlemonkey> thorus: that would be my suggestion, yes
[17:24] <scuttlemonkey> maxt: lets try to keep pastes in pastebin or public if we can...helps when other people come along and google through the logs
[17:24] <scuttlemonkey> maxt: that said, looking now
[17:24] <maxt> okay, sorry
[17:24] <scuttlemonkey> no worries
[17:26] <maxt> http://pastebin.com/XsUJhcKH
[17:26] <thorus> scuttlemonkey: do you know if I can run the mon on the proxmox?
[17:27] <maxt> looks like no peers
[17:27] * alram (~alram@38.122.20.226) has joined #ceph
[17:28] <scuttlemonkey> thorus: should be able to, mons aren't actually in the data path
[17:28] <scuttlemonkey> maxt: yeah, and osd tree is still ignorant of osd.122?
[17:28] <maxt> yes
[17:29] <scuttlemonkey> ok, take a look at the mon log to see if you're getting anything
[17:29] <scuttlemonkey> it is sending the boot message to the mon but not getting anything back
[17:29] <scuttlemonkey> should get an osd_map message back
[17:30] <scuttlemonkey> if you don't find anything you may need to turn up mon logging
[17:30] <scuttlemonkey> ceph mon tell \* injectargs '--debug-mon 20 --debug-ms 1'
[17:30] * houkouonchi-work (~linux@12.248.40.138) Quit (Ping timeout: 480 seconds)
[17:31] <maxt> only this:
[17:31] <maxt> 2013-03-27 18:21:13.665419 7fee117b5700 1 mon.b@0(leader).osd e414 boot from osd.122 >= max_osd 122
[17:31] <scuttlemonkey> lets turn up logging and see
[17:32] <maxt> restart osd.122?
[17:32] * BillK (~BillK@124.148.201.30) has joined #ceph
[17:33] <scuttlemonkey> no, turn up logging on the mon
[17:33] <scuttlemonkey> ceph mon tell \* injectargs '--debug-mon 20 --debug-ms 1'
[17:33] <maxt> this is done
[17:33] <scuttlemonkey> ahh, yeah, you can try restarting osd then
[17:33] <scuttlemonkey> see what it kicks up
[17:33] * eschnou (~eschnou@85.234.217.115.static.edpnet.net) Quit (Remote host closed the connection)
[17:35] <maxt> http://pastebin.com/28xuDmnK
[17:36] <maxt> sorry
[17:36] <maxt> this is all for osd.122
[17:37] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) Quit (Quit: Leaving.)
[17:37] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) has joined #ceph
[17:38] <scuttlemonkey> k, looking
[17:39] <sagewk> it says 'boot from osd.122 >= max_osd 122'
[17:39] <sagewk> can you pastebin 'ceph osd dump' ?
[17:39] <sagewk> it looks like that osd id isn't allocated..
[17:40] <sagewk> (which is sort of strange!)
[17:41] <sagewk> maxt: ^
[17:43] * BManojlovic (~steki@91.195.39.5) Quit (Quit: Ja odoh a vi sta 'ocete...)
[17:44] <maxt> done
[17:44] <maxt> http://pastebin.com/muNSMmFT
[17:49] * Vjarjadian (~IceChat77@5ad6d005.bb.sky.com) has joined #ceph
[17:49] * xiaoxi (~xiaoxiche@134.134.139.74) Quit (Remote host closed the connection)
[17:50] * ScOut3R_ (~ScOut3R@212.96.47.215) Quit (Ping timeout: 480 seconds)
[17:53] * Cube (~Cube@cpe-76-172-67-97.socal.res.rr.com) has joined #ceph
[17:55] * dpippenger (~riven@216.103.134.250) has joined #ceph
[17:56] * sleinen1 (~Adium@130.59.94.70) has joined #ceph
[17:56] <scuttlemonkey> sagewk: so it "knows" about osd.122, but it isn't registering?
[17:57] <sagewk> according to the map the osd doesn't exist. strange that it got into this state.
[17:57] <sagewk> if the disk has no data on it, i would just do ceph-deploy zapdisk HOST:DISK and then re-do the osd create
[17:58] <sagewk> it's also strange that all of the osd ids are so large..
[17:58] <sagewk> are these disks left over from a different cluster or something?
[17:58] <maxt> nope
[18:00] * tnt (~tnt@212-166-48-236.win.be) Quit (Ping timeout: 480 seconds)
[18:00] <rzerres> when i try to rbd map a format 1 image my server freezes and i have to hard reset. (kernel: 3.6.9-030609-generic, ubunto 12.04, ceph version 0.59-402-g76b296f (76b296f01fd0d337c8fc9f79013883e62146f0c6)
[18:01] <rzerres> i remember that i could map in ceph v0.57. is that a known issue?
[18:03] * houkouonchi-work (~linux@12.248.40.138) has joined #ceph
[18:04] * sleinen (~Adium@2001:620:0:25:4408:2cb8:dbe8:d545) Quit (Ping timeout: 480 seconds)
[18:04] * sleinen1 (~Adium@130.59.94.70) Quit (Ping timeout: 480 seconds)
[18:04] * jmlowe1 (~Adium@c-71-201-31-207.hsd1.in.comcast.net) has joined #ceph
[18:05] <jmlowe1> I'm looking for some help with my crush map
[18:06] <jmlowe1> http://pastebin.com/N6AiGir4
[18:06] * kincl (~kincl@0001aeba.user.oftc.net) has joined #ceph
[18:06] <jmlowe1> I want a replica in each rack
[18:07] * yehudasa (~yehudasa@2607:f298:a:607:c0af:5857:7027:84a7) Quit (Quit: Ex-Chat)
[18:07] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) Quit (Read error: Connection reset by peer)
[18:07] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) has joined #ceph
[18:08] <jmlowe1> do I need an extra step in there to select a rack?
[18:09] * chutzpah (~chutz@199.21.234.7) has joined #ceph
[18:10] * sjustlaptop (~sam@71-83-191-116.dhcp.gldl.ca.charter.com) has joined #ceph
[18:13] * terje1 (~Adium@c-67-176-69-220.hsd1.co.comcast.net) Quit (Quit: Leaving.)
[18:13] <scuttlemonkey> jmlowe: if all you want is one copy per rack, and you only need two copies (you have two racks) you should just be able to move your chooseleaf from type host to type rack
[18:14] <jmlowe1> so step take default
[18:14] <jmlowe1> step chooseleaf firstn 0 type rack
[18:14] <jmlowe1> step emit
[18:14] <scuttlemonkey> yeah
[18:15] <jmlowe1> ok, really want to get this right the first time
[18:15] <scuttlemonkey> can prod one of the experts if you want
[18:16] * tnt (~tnt@109.130.89.104) has joined #ceph
[18:16] <jmlowe1> sure, go ahead
[18:19] * l0nk (~alex@83.167.43.235) Quit (Quit: Leaving.)
[18:21] * jjgalvez (~jjgalvez@cpe-76-175-30-67.socal.res.rr.com) has joined #ceph
[18:21] * rturk-away is now known as rturk
[18:25] * BillK (~BillK@124.148.201.30) Quit (Ping timeout: 480 seconds)
[18:27] * askort (~askort@2a02:6b8:0:401:9f1:9da4:4351:82bc) Quit (Quit: Ухожу я от вас (xchat 2.4.5 или старше))
[18:30] <dmick> jmlowe1: you can always run crushtool in its "test simulation" mode to observe the behavior
[18:30] <jmlowe1> ooh, that sounds good
[18:30] <dmick> --test --output-csv, and then you get a pile of csv files
[18:30] <dmick> crushtool --help
[18:31] <dmick> (it's not extremely clear, but it works)
[18:34] <jmlowe1> ok, looks like I would expect
[18:36] <scuttlemonkey> hah, you learn something new every day...I didn't even realize we had a test dump
[18:36] <scuttlemonkey> thanks dmick
[18:38] <dmick> I found it by accident :)
[18:38] <dmick> but I'm trying to publicize it; it's useful for nontrivial CRUSH maps, which they all are :)
[18:38] <scuttlemonkey> hehe nod
[18:39] <scuttlemonkey> I'm actually hoping some enterprising Google Summer of Code student decides to pick up crush and really make it stand on its head
[18:39] <scuttlemonkey> would be neat to see new applications and/or more complex rulesets
[18:48] * jtang2 (~jtang@2001:770:10:500:3d22:b2b:cf17:51a1) Quit (Ping timeout: 480 seconds)
[18:50] <rturk> I wonder if we should suggest that people use --test in our crush map doc page
[18:51] <rturk> it's in the man page for crushtool, but maybe it would be helpful to mention there too
[18:51] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) Quit (Ping timeout: 480 seconds)
[18:52] * yehudasa (~yehudasa@2607:f298:a:607:9911:7a60:6fef:7eb5) has joined #ceph
[18:52] <scuttlemonkey> couldn't hurt I guess
[18:52] <scuttlemonkey> I'll file a ticket for john
[18:53] <yehudasa> scuttlemonkey: can you update channel title?
[18:53] * BManojlovic (~steki@fo-d-130.180.254.37.targo.rs) has joined #ceph
[18:54] * scuttlemonkey changes topic to 'v0.56.4 has been released -- http://goo.gl/kRvhC || argonaut v0.48.3 released -- http://goo.gl/80aGP || Ceph & iSCSI via tgt http://goo.gl/vKWyu'
[18:54] <yehudasa> awesome, thanks
[18:55] <scuttlemonkey> yah, thanks for reminding me
[18:55] * Cube (~Cube@cpe-76-172-67-97.socal.res.rr.com) Quit (Quit: Leaving.)
[18:55] <scuttlemonkey> I need to make a list of things to stay on top of...turns out lists are better than my brain at remembering things
[19:00] * Kioob (~kioob@2a01:e35:2432:58a0:21e:8cff:fe07:45b6) has joined #ceph
[19:04] * maxt (~sivanov@gw2.maxtelecom.bg) Quit (Ping timeout: 480 seconds)
[19:07] <sstan> Still having problems with one particular monitor (out of 3). It shuts down when it serves a clients! Fails some assert ... please see logs : http://pastebin.com/5twihiHJ
[19:09] * rzerres1 (~ralf@bermuda.daywalker-studios.de) has joined #ceph
[19:10] * Vjarjadian (~IceChat77@5ad6d005.bb.sky.com) Quit (Quit: Why is the alphabet in that order? Is it because of that song?)
[19:11] <joshd> nz_monkey_: yes, it should be ready today, but it requires a patch to qemu too unfortunately
[19:15] * leseb (~Adium@3.46-14-84.ripe.coltfrance.com) Quit (Quit: Leaving.)
[19:15] * rzerres (~ralf.zerr@bermuda.daywalker-studios.de) Quit (Ping timeout: 480 seconds)
[19:20] * Cube (~Cube@12.248.40.138) has joined #ceph
[19:25] <wido> sagewk: you around?
[19:25] <wido> sagewk: never mind, just saw your update
[19:25] <wido> on #4556
[19:26] * rzerres1 (~ralf@bermuda.daywalker-studios.de) has left #ceph
[19:28] * dosaboy (~gizmo@faun.canonical.com) Quit (Quit: Leaving.)
[19:28] <sagewk> wido: can you get logs?
[19:28] <sagewk> wondering what in the workload is triggering it..
[19:31] * jjgalvez (~jjgalvez@cpe-76-175-30-67.socal.res.rr.com) Quit (Quit: Leaving.)
[19:31] <wido> sagewk: debug osd?
[19:31] <wido> Or ms?
[19:31] <sagewk> debug osd = 10, debug ms = 1 should do it
[19:31] <sagewk> what is the workload?
[19:31] <wido> sagewk: The workload is a CloudStack cluster with 2 hypervisors running about 10 VMs
[19:32] <sagewk> is it the bobtail version of librbd?
[19:32] <sagewk> oh, maybe it is the libvirt rbd pool code.. we don't test that
[19:32] <wido> sagewk: 0.56-1precise
[19:33] <wido> both Qemu and libvirt are compiled against that librbd
[19:33] <sagewk> hmm
[19:33] <wido> So, I'll start with those debug values
[19:33] <wido> It will take about 1hr to crash all OSDs
[19:33] <sagewk> 0.56 itself had broken comptatibility
[19:33] <sagewk> those compiling against it is probably ok, as long as the .so itself is newer
[19:34] <sagewk> cool
[19:34] <wido> I know that 0.56 had issues, ran into those. But that's why it's at least 0.56.1
[19:35] <sagewk> they compile against 0.56-1precise though?
[19:35] <sagewk> or 0.56.1-1precise?
[19:36] <wido> against 0.56-1precise
[19:36] <wido> hmm, that rings a bell
[19:36] <wido> It's indeed 0.56 on that machine, not 0.56.1
[19:37] <wido> sagewk: Shall I kill the clients prior to testing?
[19:38] <sagewk> that might be it...
[19:38] <sagewk> i'd be interested in seeing hte logs even with broken clients just to be sure we understand what is happening
[19:39] <sagewk> just need to see the log msgs from teh request coming in right before the crash.
[19:39] <wido> Ok, I'll start them now. I'll be back in about an hour with logs
[19:42] * stacker100 (~stacker66@152.pool85-58-180.dynamic.orange.es) Quit (Ping timeout: 480 seconds)
[19:48] * nwat (~Adium@eduroam-233-33.ucsc.edu) has joined #ceph
[19:51] <Kioob> Hi
[19:51] <Kioob> 2013-03-27 19:49:47.596770 7f489f7f2700 0 log [WRN] : 17 slow requests, 1 included below; oldest blocked for > 200.206025 secs
[19:51] <Kioob> 2013-03-27 19:49:47.596790 7f489f7f2700 0 log [WRN] : slow request 60.743144 seconds old, received at 2013-03-27 19:48:46.853585: osd_op(client.4721.1:805670817 rb.0.1341.238e1f29.000000000160 [write 196608~4096] 3.300caf33 RETRY snapc 1a9a=[1a9a,1a6d,1a3f,1a11,19e4,19b5,1988,1959,192c,18fe,18d0,18a6,1884,1862,1840,181e,17fc,17da,17b8,1796,1774,174b,171d,16ef,16c1,1693,1665,1638,1609,15dd,15b5,1590,1562,1538,150f,14e1,14b3,1486,1464,1442,
[19:51] <Kioob> I have that in logs
[19:51] <Kioob> the huge bloc �snapc�, is about creating snapshots ?
[19:53] <Kioob> so.... snapshot create is async, and if I ask a snapshot of all VM, I �overload� disks ?
[19:55] * jjgalvez (~jjgalvez@12.248.40.138) has joined #ceph
[20:08] * mcclurmc_laptop (~mcclurmc@firewall.ctxuk.citrix.com) Quit (Ping timeout: 480 seconds)
[20:08] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) has joined #ceph
[20:18] <Kioob> I have 2 mons (on 5) which complains about one other mon : mon.e@2(peon).paxos(osdmap active c 28056..32459) lease_expire from mon.0 10.0.0.1:6789/0 is 0.796211 seconds in the past; mons are laggy or clocks are too skewed
[20:18] <Kioob> I verified clocked
[20:19] <Kioob> so I suppose that mon is lagging, but why other mons doesn't complain ?
[20:24] <wido> sagewk: Ok if I drop the files on /home/cephdrop at ceph.com?
[20:24] <sagewk> sure
[20:24] <wido> logs are 130MB gzip
[20:25] <wido> Ok, I'll do
[20:35] * nwat (~Adium@eduroam-233-33.ucsc.edu) Quit (Quit: Leaving.)
[20:35] <wido> sagewk: Just posted the logs in 4556
[20:36] <sagewk> wido k thanks
[20:49] <nz_monkey_> joshd: Ok, best to wait for 0.61 then I guess
[20:52] <sagewk> wido: is 2a00:f10:11b:fe56:38fd:1721:fb19:b57d the one with 0.56 ?
[20:52] * ivotron (~ivotron@adsl-76-254-17-170.dsl.pltn13.sbcglobal.net) has joined #ceph
[20:53] * aliguori (~anthony@cpe-70-112-157-87.austin.res.rr.com) Quit (Read error: Connection reset by peer)
[20:54] * aliguori (~anthony@cpe-70-112-157-87.austin.res.rr.com) has joined #ceph
[20:54] <sagewk> wido: yeah, it looks like a v0.56 client.
[20:54] <wido> sagewk: Not sure. Doesn't seem to be online that IP. It's not on one of my clients
[20:54] <wido> it's my IP range though
[20:57] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) has joined #ceph
[21:00] * jeffv (~jeffv@2607:fad0:32:a02:990f:d0f4:34e5:a977) Quit (Quit: Leaving.)
[21:01] * jeffv (~jeffv@2607:fad0:32:a02:4840:6bd6:162:515a) has joined #ceph
[21:02] <sagewk> wido: preparing a fix. the real problem on your cluster tho is that 0.56 client.. witht eh fix it'll get EINVAL (instead of making ceph-osd crash)
[21:04] <wido> sagewk: Yes I know. But still, you shouldn't be able to cradh a osd that easy
[21:04] <sagewk> yeah :) fixing that part
[21:04] <wido> one rogue client still present could kill your cluster
[21:04] <wido> I just upgraded the clients :)
[21:05] * jtang1 (~jtang@79.97.135.214) has joined #ceph
[21:14] <wido> btw, good news. New Atom CPUs with 16GB memory support coming up :)
[21:14] <wido> SuperMicro should have stuff later this year
[21:14] * Mastema (~luci@pool-173-48-36-205.bstnma.fios.verizon.net) Quit (Ping timeout: 480 seconds)
[21:15] <sagewk> nice
[21:15] <dwm37> wido: Intervestink.
[21:16] * nwat (~Adium@c-50-131-197-174.hsd1.ca.comcast.net) has joined #ceph
[21:23] <sagewk> wido: pushed bobtail-4556 which has the backported fixes. going to make sure things are behaving in master first before putting them in the bobtail branch itself.
[21:23] <sagewk> if you want to downgrade that client and test the branch, that'd be nice.. otherwise no worries
[21:27] <sagewk> and thanks for helping track this down!
[21:30] * nhorman (~nhorman@2001:470:8:a08:7aac:c0ff:fec2:933b) Quit (Quit: Leaving)
[21:32] * sjustlaptop (~sam@71-83-191-116.dhcp.gldl.ca.charter.com) Quit (Ping timeout: 480 seconds)
[21:37] <PerlStalker> Does anyone have logstash patterns for ceph logs?
[21:40] * scalability-junk (uid6422@id-6422.tooting.irccloud.com) Quit (Remote host closed the connection)
[21:42] * sagewk (~sage@2607:f298:a:607:2055:f0c4:52ca:ba9) Quit (Ping timeout: 480 seconds)
[21:44] <loicd> sjust: Hi ! If you have time I would very much appreciate a review of the lfn_get_name unit tests. https://github.com/ceph/ceph/pull/161 .
[21:47] * sagewk (~sage@2607:f298:a:607:a54f:880a:b8a2:844c) has joined #ceph
[21:48] <loicd> It turned out to be easy to test despite the fact that it's a private method. code.google.com/p/googletest/wiki/AdvancedGuide#Testing_Private_Code suggests ways to help testing private code. The best way would be to have separate files for the private methods / data member. But it would probably be overkill to refactor the code just for this reason.
[21:49] * verwilst (~verwilst@dD5769628.access.telenet.be) has joined #ceph
[21:49] * sjust-phone (~sjust@m990436d0.tmodns.net) has joined #ceph
[21:49] * mcclurmc_laptop (~mcclurmc@cpc10-cmbg15-2-0-cust205.5-4.cable.virginmedia.com) has joined #ceph
[21:57] * nwat (~Adium@c-50-131-197-174.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[22:03] * sjustlaptop (~sam@m990436d0.tmodns.net) has joined #ceph
[22:05] * Tribaal (uid3081@hillingdon.irccloud.com) Quit (Remote host closed the connection)
[22:15] * Vjarjadian (~IceChat77@5ad6d005.bb.sky.com) has joined #ceph
[22:29] * Tribaal (uid3081@id-3081.richmond.irccloud.com) has joined #ceph
[22:35] * loicd (~loic@74-94-156-210-NewEngland.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[22:38] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[22:40] * scalability-junk (uid6422@id-6422.hillingdon.irccloud.com) has joined #ceph
[22:43] <absynth> so, as it turns out, you shouldn't try to transport 300gbps of dns traffic via LINX and DE-CIX
[22:44] <Meths> lol
[22:44] <absynth> _slightly_ unsolicited dns traffic
[22:45] <dmick> absynth: I suppose it depends on your goal
[22:45] <absynth> if your goal is taking down the internet, you might eventually succeed
[22:45] <absynth> if your goal is *anything* else, you will fail
[22:46] * jskinner (~jskinner@69.170.148.179) Quit (Remote host closed the connection)
[22:47] <dmick> I think the goal was likely "huhhhuhhuhhh let's f*^* sh*& up"
[22:48] <absynth> specifically, let's f*** spamhaus up
[22:48] <absynth> didn't work though, and cloudflare now officially has the biggest, err, tool.
[22:48] <dmick> oh?
[22:49] <absynth> http://blog.cloudflare.com/the-ddos-that-almost-broke-the-internet
[22:49] <absynth> the title might be slightly exaggerated, but 300gbps is quite a lot
[22:50] <Meths> Only slightly with the dip in the LINX graph.
[22:50] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) Quit (Read error: Connection reset by peer)
[22:50] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) has joined #ceph
[22:51] <absynth> a couple people on the linx list were slightly agitated though
[22:51] <absynth> i only read the filtered drama on the decix lists
[22:53] * mikedawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[22:56] * chftosf (uid7988@id-7988.hillingdon.irccloud.com) has joined #ceph
[22:57] <nhm> ugh, AMD is releasing graphics cards for "cloud gaming".
[22:57] <dmick> dibs on Lando Calrissian
[22:57] <jmlowe1> yeah, 'cause you know how network speeds keep up with moore's observation
[22:58] * aliguori (~anthony@cpe-70-112-157-87.austin.res.rr.com) Quit (Remote host closed the connection)
[22:59] <absynth> in other news: german dsl carriers are pondering to re-introduce bandwidth limits on consumer lines.
[22:59] <absynth> there goes europe's biggest gaming market, AMD.
[23:01] * drokita1 (~drokita@199.255.228.128) has joined #ceph
[23:03] * sjustlaptop (~sam@m990436d0.tmodns.net) Quit (Ping timeout: 480 seconds)
[23:06] * eschnou (~eschnou@227.159-201-80.adsl-dyn.isp.belgacom.be) has joined #ceph
[23:07] * drokita (~drokita@199.255.228.128) Quit (Ping timeout: 480 seconds)
[23:07] <tchmnkyz> ok so how can i correct this.... 6 pgs inconsistent; 6 scrub errors
[23:09] <jmlowe1> tchmnkyz: how many replicas, what filesystem did you use on your osd's?
[23:09] * drokita1 (~drokita@199.255.228.128) Quit (Ping timeout: 480 seconds)
[23:10] <dmick> http://ceph.com/docs/master/rados/operations/troubleshooting-osd/#troubleshooting-pg-errors
[23:10] * BillK (~BillK@58-7-223-82.dyn.iinet.net.au) has joined #ceph
[23:11] <tchmnkyz> 2 replicas and i use xfs
[23:13] <jmlowe1> I've got nothing, do what you can with dmick's link
[23:13] * mcclurmc_laptop (~mcclurmc@cpc10-cmbg15-2-0-cust205.5-4.cable.virginmedia.com) Quit (Ping timeout: 480 seconds)
[23:17] * sjustlaptop (~sam@71-83-191-116.dhcp.gldl.ca.charter.com) has joined #ceph
[23:18] <jmlowe1> tchmnkyz: dmesg have anything about drive failures?
[23:19] * sjust-phone (~sjust@m990436d0.tmodns.net) Quit (Ping timeout: 480 seconds)
[23:21] * timmclau_ (~timmclaug@69.170.148.179) Quit (Remote host closed the connection)
[23:22] * eschnou (~eschnou@227.159-201-80.adsl-dyn.isp.belgacom.be) Quit (Ping timeout: 480 seconds)
[23:23] * JohansGlock (~quassel@kantoor.transip.nl) Quit (Quit: No Ping reply in 180 seconds.)
[23:23] * JohansGlock (~quassel@kantoor.transip.nl) has joined #ceph
[23:25] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[23:25] <tchmnkyz> ok i was able to recover them
[23:25] * Mastema (~luci@pool-173-48-36-205.bstnma.fios.verizon.net) has joined #ceph
[23:25] <tchmnkyz> jmlowe1: no i know that osd is having raid problems though
[23:25] <tchmnkyz> my osd's are on 24 drive raid 50 arrays
[23:31] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[23:35] * loicd (~loic@209.117.47.248) has joined #ceph
[23:35] * loicd (~loic@209.117.47.248) Quit ()
[23:40] * The_Bishop (~bishop@2001:470:50b6:0:1475:dc56:2e23:e501) Quit (Quit: Wer zum Teufel ist dieser Peer? Wenn ich den erwische dann werde ich ihm mal die Verbindung resetten!)
[23:46] * verwilst (~verwilst@dD5769628.access.telenet.be) Quit (Quit: Ex-Chat)
[23:49] * PerlStalker (~PerlStalk@72.166.192.70) Quit (Quit: ...)
[23:58] * jlogan1 (~Thunderbi@2600:c00:3010:1:8527:f7:8cac:c375) Quit (Ping timeout: 480 seconds)

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.