#ceph IRC Log

Index

IRC Log for 2014-04-09

Timestamps are in GMT/BST.

[0:04] * jtang1 (~jtang@80.111.79.253) Quit (Quit: Leaving.)
[0:05] * zidarsk8 (~zidar@89-212-142-10.dynamic.t-2.net) has joined #ceph
[0:16] * japuzzo (~japuzzo@ool-4570886e.dyn.optonline.net) Quit (Quit: Leaving)
[0:16] * BManojlovic (~steki@cable-94-189-160-252.dynamic.sbb.rs) Quit (Quit: Ja odoh a vi sta 'ocete...)
[0:20] * analbeard (~shw@host31-53-108-38.range31-53.btcentralplus.com) has joined #ceph
[0:24] * analbeard (~shw@host31-53-108-38.range31-53.btcentralplus.com) has left #ceph
[0:34] * imriz (~imriz@5.29.200.177) Quit (Remote host closed the connection)
[0:35] * BillK (~BillK-OFT@106-69-38-152.dyn.iinet.net.au) has joined #ceph
[0:38] * Sysadmin88 (~IceChat77@05452a50.skybroadband.com) has joined #ceph
[0:39] * Shmouel1 (~Sam@fny94-12-83-157-27-95.fbx.proxad.net) Quit (Read error: Operation timed out)
[0:39] * The_Bishop (~bishop@f055013243.adsl.alicedsl.de) has joined #ceph
[0:49] * joao (~joao@a95-92-33-54.cpe.netcabo.pt) Quit (Ping timeout: 480 seconds)
[0:50] * thuc (~thuc@c-71-198-202-49.hsd1.ca.comcast.net) has joined #ceph
[0:52] * jeff-YF (~jeffyf@67.23.117.122) Quit (Quit: jeff-YF)
[0:52] * Cube (~Cube@66.87.66.39) Quit (Read error: Connection reset by peer)
[0:54] * Shmouel (~Sam@fny94-12-83-157-27-95.fbx.proxad.net) has joined #ceph
[0:56] * mtanski (~mtanski@69.193.178.202) Quit (Ping timeout: 480 seconds)
[0:56] * diegows (~diegows@190.190.5.238) Quit (Ping timeout: 480 seconds)
[0:57] <kiwigeraint> Hi guys
[0:57] <kiwigeraint> is there an opposite to injectargs ?
[0:57] * zack_dolby (~textual@p852cae.tokynt01.ap.so-net.ne.jp) Quit (Quit: My MacBook has gone to sleep. ZZZzzz???)
[0:57] <kiwigeraint> i have a mon thats logging like crazy and want to stop it :)
[0:58] * lofejndif (~lsqavnbok@bouazizi.torservers.net) has joined #ceph
[0:58] * thuc (~thuc@c-71-198-202-49.hsd1.ca.comcast.net) Quit (Read error: Operation timed out)
[1:01] * Cube (~Cube@66.87.66.39) has joined #ceph
[1:01] * ircolle (~Adium@2601:1:8380:2d9:48c7:43ce:d288:498b) Quit (Quit: Leaving.)
[1:04] <gregsfortytwo> you might try doing it again with a different logging value ;)
[1:07] * zidarsk8 (~zidar@89-212-142-10.dynamic.t-2.net) has left #ceph
[1:08] <sjustwork> sage: ping
[1:09] <kiwigeraint> @gregsfortytwo thanks :)
[1:09] <cephalobot> kiwigeraint: Error: "gregsfortytwo" is not a valid command.
[1:09] <kiwigeraint> also seeing this on all my osds :"2014-04-09 00:06:16.436087 7fbd87d91700 0 -- 192.168.54.19:6804/30813 >> 68.68.97.160:0/74936070 pipe(0x825b280 sd=184 :6804 s=0 pgs=0 cs=0 l=0 c=0x807bc60).accept peer addr is really 68.68.97.160:0/74936070 (socket is 68.68.97.160:48936/0)"
[1:14] * Cube (~Cube@66.87.66.39) Quit (Read error: Connection reset by peer)
[1:22] * zack_dolby (~textual@e0109-114-22-6-65.uqwimax.jp) has joined #ceph
[1:26] <pmatulis> n1md4: welcome
[1:26] * thb (~me@0001bd58.user.oftc.net) Quit (Ping timeout: 480 seconds)
[1:28] * The_Bishop (~bishop@f055013243.adsl.alicedsl.de) Quit (Quit: Wer zum Teufel ist dieser Peer? Wenn ich den erwische dann werde ich ihm mal die Verbindung resetten!)
[1:32] * thuc (~thuc@c-71-198-202-49.hsd1.ca.comcast.net) has joined #ceph
[1:32] * The_Bishop (~bishop@f055013243.adsl.alicedsl.de) has joined #ceph
[1:33] * zack_dolby (~textual@e0109-114-22-6-65.uqwimax.jp) Quit (Quit: My MacBook has gone to sleep. ZZZzzz???)
[1:34] * Cube (~Cube@66-87-66-39.pools.spcsdns.net) has joined #ceph
[1:40] * thuc (~thuc@c-71-198-202-49.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[1:46] * lurbs (user@uber.geek.nz) Quit (Quit: Changing server)
[1:48] * lurbs (user@uber.geek.nz) has joined #ceph
[1:49] * zack_dolby (~textual@e0109-114-22-6-65.uqwimax.jp) has joined #ceph
[1:53] * lofejndif (~lsqavnbok@83TAAILOW.tor-irc.dnsbl.oftc.net) Quit (Quit: gone)
[1:56] * kevincox (~kevincox@4.s.kevincox.ca) Quit (Quit: Bye)
[1:58] * kevincox (~kevincox@4.s.kevincox.ca) has joined #ceph
[2:00] * shang (~ShangWu@220-135-203-169.HINET-IP.hinet.net) has joined #ceph
[2:08] * JoeGruher (~JoeGruher@jfdmzpr04-ext.jf.intel.com) Quit (Remote host closed the connection)
[2:09] * rmoe (~quassel@12.164.168.117) Quit (Ping timeout: 480 seconds)
[2:10] * zidarsk8 (~zidar@tm.78.153.58.217.dc.cable.static.telemach.net) has joined #ceph
[2:11] * alfredodeza (~alfredode@198.206.133.89) has joined #ceph
[2:11] * sjustwork (~sam@2607:f298:a:607:3570:d0e7:4da5:f7bd) Quit (Quit: Leaving.)
[2:12] * LeaChim (~LeaChim@host86-162-1-71.range86-162.btcentralplus.com) Quit (Read error: Operation timed out)
[2:12] * zidarsk8 (~zidar@tm.78.153.58.217.dc.cable.static.telemach.net) has left #ceph
[2:19] * mtanski (~mtanski@cpe-72-229-51-156.nyc.res.rr.com) has joined #ceph
[2:20] * rmoe (~quassel@173-228-89-134.dsl.static.sonic.net) has joined #ceph
[2:25] * andreask (~andreask@h081217016175.dyn.cm.kabsi.at) Quit (Quit: Leaving.)
[2:28] * thuc (~thuc@c-71-198-202-49.hsd1.ca.comcast.net) has joined #ceph
[2:30] * alram (~alram@cpe-76-167-50-51.socal.res.rr.com) Quit (Quit: leaving)
[2:31] * mtanski (~mtanski@cpe-72-229-51-156.nyc.res.rr.com) Quit (Quit: mtanski)
[2:32] * eternaleye (~eternaley@50.245.141.73) Quit (Quit: ZNC - http://znc.in)
[2:33] * eternaleye (~eternaley@50.245.141.73) has joined #ceph
[2:36] * thuc (~thuc@c-71-198-202-49.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[2:50] * mtanski (~mtanski@cpe-72-229-51-156.nyc.res.rr.com) has joined #ceph
[2:56] * sputnik13 (~sputnik13@207.8.121.241) Quit (Ping timeout: 480 seconds)
[3:00] * jjgalvez (~jjgalvez@ip98-167-16-160.lv.lv.cox.net) Quit (Quit: Leaving.)
[3:00] * AfC (~andrew@101.119.14.247) has joined #ceph
[3:06] * kiwigeraint (~kiwigerai@208.72.139.54) Quit (Remote host closed the connection)
[3:07] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit (Quit: Leaving.)
[3:08] * AfC (~andrew@101.119.14.247) Quit (Quit: Leaving.)
[3:10] * JeffK (~Jeff@38.99.52.10) Quit (Quit: JeffK)
[3:10] * kiwigeraint (~kiwigerai@208.72.139.54) has joined #ceph
[3:12] * Boltsky (~textual@office.deviantart.net) Quit (Ping timeout: 480 seconds)
[3:16] * tserong (~tserong@203-57-208-132.dyn.iinet.net.au) Quit (Quit: Leaving)
[3:16] * mjevans (~mje@209.141.34.79) Quit (Remote host closed the connection)
[3:17] * Jahkeup (Jahkeup@gemini.ca.us.panicbnc.net) Quit (Quit: PanicBNC - https://PanicBNC.net)
[3:17] * jjgalvez (~jjgalvez@ip98-167-16-160.lv.lv.cox.net) has joined #ceph
[3:17] * sjusthm (~sam@24-205-43-60.dhcp.gldl.ca.charter.com) has joined #ceph
[3:19] * bandrus (~Adium@12.104.140.22) has joined #ceph
[3:20] * bandrus (~Adium@12.104.140.22) Quit ()
[3:20] * Jahkeup (Jahkeup@gemini.ca.us.panicbnc.net) has joined #ceph
[3:21] * xarses (~andreww@12.164.168.117) Quit (Ping timeout: 480 seconds)
[3:31] * mtanski (~mtanski@cpe-72-229-51-156.nyc.res.rr.com) Quit (Quit: mtanski)
[3:31] * tracphil_ (~tracphil@pool-71-163-168-251.washdc.fios.verizon.net) has joined #ceph
[3:32] * tserong (~tserong@203-57-208-132.dyn.iinet.net.au) has joined #ceph
[3:33] * zerick (~eocrospom@190.187.21.53) Quit (Remote host closed the connection)
[3:34] * sjusthm (~sam@24-205-43-60.dhcp.gldl.ca.charter.com) Quit (Quit: Leaving.)
[3:35] * shang (~ShangWu@220-135-203-169.HINET-IP.hinet.net) Quit (Ping timeout: 480 seconds)
[3:36] * JCL (~JCL@c-24-23-166-139.hsd1.ca.comcast.net) has joined #ceph
[3:36] * kiwigeraint (~kiwigerai@208.72.139.54) Quit (Remote host closed the connection)
[3:36] * t0rn (~ssullivan@c-24-11-198-35.hsd1.mi.comcast.net) has joined #ceph
[3:39] * kiwigeraint (~kiwigerai@208.72.139.54) has joined #ceph
[3:39] * JCL2 (~JCL@c-24-23-166-139.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[3:39] * thuc (~thuc@c-71-198-202-49.hsd1.ca.comcast.net) has joined #ceph
[3:40] * dpippenger (~riven@66-192-9-78.static.twtelecom.net) Quit (Quit: Leaving.)
[3:47] * haomaiwa_ (~haomaiwan@118.186.133.135) Quit (Remote host closed the connection)
[3:47] * Boltsky (~textual@cpe-198-72-138-106.socal.res.rr.com) has joined #ceph
[3:47] * haomaiwang (~haomaiwan@219-87-173-15.static.tfn.net.tw) has joined #ceph
[3:47] * thuc (~thuc@c-71-198-202-49.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[3:49] * alfredodeza (~alfredode@198.206.133.89) Quit (Ping timeout: 480 seconds)
[3:57] * shang (~ShangWu@220-135-203-169.HINET-IP.hinet.net) has joined #ceph
[3:57] * shang (~ShangWu@220-135-203-169.HINET-IP.hinet.net) Quit ()
[3:58] * shang (~ShangWu@220-135-203-169.HINET-IP.hinet.net) has joined #ceph
[3:58] * valeech (~valeech@pool-71-171-123-210.clppva.fios.verizon.net) has joined #ceph
[4:00] * talonisx (~talonisx@pool-108-18-97-131.washdc.fios.verizon.net) has joined #ceph
[4:06] * haomaiwa_ (~haomaiwan@117.79.232.255) has joined #ceph
[4:08] * tracphil_ (~tracphil@pool-71-163-168-251.washdc.fios.verizon.net) Quit (Read error: Connection reset by peer)
[4:09] * tracphil_ (~tracphil@pool-71-163-168-251.washdc.fios.verizon.net) has joined #ceph
[4:09] * alfredodeza (~alfredode@198.206.133.89) has joined #ceph
[4:11] * tracphil_ (~tracphil@pool-71-163-168-251.washdc.fios.verizon.net) Quit ()
[4:11] * tracphil_ (~tracphil@pool-71-163-168-251.washdc.fios.verizon.net) has joined #ceph
[4:13] * haomaiwang (~haomaiwan@219-87-173-15.static.tfn.net.tw) Quit (Ping timeout: 480 seconds)
[4:16] * BillK (~BillK-OFT@106-69-38-152.dyn.iinet.net.au) Quit (Ping timeout: 480 seconds)
[4:18] * BillK (~BillK-OFT@106-68-27-227.dyn.iinet.net.au) has joined #ceph
[4:26] * xarses (~andreww@c-24-23-183-44.hsd1.ca.comcast.net) has joined #ceph
[4:30] * AfC (~andrew@2407:7800:400:107f:2ad2:44ff:fe08:a4c) has joined #ceph
[4:45] * kiwigeraint (~kiwigerai@208.72.139.54) Quit (Remote host closed the connection)
[5:01] * Pedras1 (~Adium@c-67-188-26-20.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[5:03] * MACscr (~Adium@c-50-158-183-38.hsd1.il.comcast.net) Quit (Quit: Leaving.)
[5:03] * Vacum (~vovo@i59F7A9BB.versanet.de) has joined #ceph
[5:10] * Vacum_ (~vovo@i59F797BB.versanet.de) Quit (Ping timeout: 480 seconds)
[5:13] * The_Bishop (~bishop@f055013243.adsl.alicedsl.de) Quit (Ping timeout: 480 seconds)
[5:14] * JoeGruher (~JoeGruher@134.134.137.73) has joined #ceph
[5:15] * tracphil (~tracphil@130.14.71.217) Quit (Read error: Connection reset by peer)
[5:16] * Cube (~Cube@66-87-66-39.pools.spcsdns.net) Quit (Ping timeout: 480 seconds)
[5:17] * AfC (~andrew@2407:7800:400:107f:2ad2:44ff:fe08:a4c) Quit (Quit: Leaving.)
[5:20] * kiwigeraint (~kiwigerai@208.72.139.54) has joined #ceph
[5:25] * talonisx (~talonisx@pool-108-18-97-131.washdc.fios.verizon.net) Quit (Quit: KVIrc 4.2.0 Equilibrium http://www.kvirc.net/)
[5:25] * rmoe (~quassel@173-228-89-134.dsl.static.sonic.net) Quit (Quit: No Ping reply in 180 seconds.)
[5:33] * shang (~ShangWu@220-135-203-169.HINET-IP.hinet.net) Quit (Ping timeout: 480 seconds)
[5:33] * rmoe (~quassel@173-228-89-134.dsl.static.sonic.net) has joined #ceph
[5:38] * JoeGruher (~JoeGruher@134.134.137.73) Quit (Remote host closed the connection)
[5:55] * alfredodeza (~alfredode@198.206.133.89) Quit (Ping timeout: 480 seconds)
[6:08] * ido (~ido@00014f21.user.oftc.net) Quit (Quit: leaving)
[6:09] * samuel (~samuelzav@189.196.64.219) has joined #ceph
[6:10] * thuc (~thuc@c-71-198-202-49.hsd1.ca.comcast.net) has joined #ceph
[6:10] * samuel (~samuelzav@189.196.64.219) Quit ()
[6:13] * wschulze (~wschulze@cpe-72-229-37-201.nyc.res.rr.com) Quit (Quit: Leaving.)
[6:18] * thuc (~thuc@c-71-198-202-49.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[6:24] * flaxy (~afx@78.130.174.164) Quit (Ping timeout: 480 seconds)
[6:29] * alfredodeza (~alfredode@198.206.133.89) has joined #ceph
[6:32] * flaxy (~afx@78.130.174.164) has joined #ceph
[6:42] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) has joined #ceph
[6:57] * rdas (~rdas@nat-pool-pnq-t.redhat.com) has joined #ceph
[6:59] * imriz (~imriz@5.29.200.177) has joined #ceph
[7:02] <imriz> Hi, few of my OSDs are crashing with "Internal error xfs_da_do_buf(2) at line 2136 of file fs/xfs/xfs_da_btree.c. Caller 0xffffffffa01bb06a". I am running kernel 2.6.32-431.5.1.el6.x86_64. Any idea?
[7:02] * thuc (~thuc@c-71-198-202-49.hsd1.ca.comcast.net) has joined #ceph
[7:06] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[7:08] * raso (~raso@deb-multimedia.org) Quit (Quit: WeeChat 0.4.3)
[7:09] * raso (~raso@deb-multimedia.org) has joined #ceph
[7:10] * thuc (~thuc@c-71-198-202-49.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[7:13] * AfC (~andrew@2407:7800:400:107f:6e88:14ff:fe33:2a9c) has joined #ceph
[7:28] * carif (~mcarifio@hotelaramis2.pck.nerim.net) has joined #ceph
[7:30] * tiger (~textual@58.213.102.114) has joined #ceph
[7:31] * cynosure (~vilobhmm@nat-dip28-wl-b.cfw-a-gci.corp.yahoo.com) Quit (Quit: cynosure)
[7:31] * tiger (~textual@58.213.102.114) Quit ()
[7:36] * carif (~mcarifio@hotelaramis2.pck.nerim.net) Quit (Quit: Ex-Chat)
[7:37] * AfC (~andrew@2407:7800:400:107f:6e88:14ff:fe33:2a9c) Quit (Quit: Leaving.)
[7:37] * AfC (~andrew@2407:7800:400:107f:2ad2:44ff:fe08:a4c) has joined #ceph
[7:40] * valeech (~valeech@pool-71-171-123-210.clppva.fios.verizon.net) Quit (Quit: valeech)
[7:41] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[7:49] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) has joined #ceph
[7:54] <imriz> when filestore_xattr_use_omap is set to true, does it still consider filestore_max_inline_xattr_size or will it always use omap?
[7:57] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[8:00] * carif (~mcarifio@hotelaramis2.pck.nerim.net) has joined #ceph
[8:05] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) has joined #ceph
[8:07] * xarses (~andreww@c-24-23-183-44.hsd1.ca.comcast.net) Quit (Read error: Operation timed out)
[8:12] * shang (~ShangWu@175.41.48.77) has joined #ceph
[8:12] * thuc (~thuc@c-71-198-202-49.hsd1.ca.comcast.net) has joined #ceph
[8:13] * toabctl (~toabctl@toabctl.de) Quit (Quit: WeeChat 0.3.7)
[8:13] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[8:13] * toabctl (~toabctl@toabctl.de) has joined #ceph
[8:20] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) has joined #ceph
[8:20] * thuc (~thuc@c-71-198-202-49.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[8:29] * blahnana (~bman@us1.blahnana.com) Quit (Ping timeout: 480 seconds)
[8:29] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[8:30] * blahnana (~bman@us1.blahnana.com) has joined #ceph
[8:31] * tiger (~textual@58.213.102.114) has joined #ceph
[8:32] * thb (~me@2a02:2028:5b:1e80:6267:20ff:fec9:4e40) has joined #ceph
[8:32] * thb is now known as Guest5944
[8:33] * Guest5944 is now known as thb
[8:33] * Cube (~Cube@66.87.65.44) has joined #ceph
[8:34] * tiger (~textual@58.213.102.114) Quit ()
[8:36] * asteriod (~asteriod@123-3-160-83.static.dsl.dodo.com.au) has joined #ceph
[8:37] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) has joined #ceph
[8:38] * b0e (~aledermue@37-5-55-13-dynip.superkabel.de) has joined #ceph
[8:39] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit (Quit: My MacBook has gone to sleep. ZZZzzz???)
[8:40] * asteriod (~asteriod@123-3-160-83.static.dsl.dodo.com.au) Quit (Quit: Leaving)
[8:41] <Fruit> imriz: sounds like your XFS has trouble. is your disk ok? if it is, it might be the kernel that has a genuine bug. if so, call RH
[8:42] <imriz> Fruit, it happened on few different disks
[8:42] <imriz> on multiple machines
[8:42] * carif (~mcarifio@hotelaramis2.pck.nerim.net) Quit (Read error: Operation timed out)
[8:42] <imriz> the fact it is related to xattr is whats alarming me
[8:42] <imriz> http://tracker.ceph.com/issues/6143
[8:42] <Fruit> well that is easily remedied
[8:43] <Fruit> switch to omap attributes
[8:43] <imriz> I already have omap :)
[8:43] <imriz> it seems to be ignored?
[8:43] <imriz> ]# ceph --admin-daemon /var/run/ceph/ceph-osd.34.asok config show |grep omap
[8:43] <imriz> "filestore_debug_omap_check": "false",
[8:44] <Fruit> hmm
[8:44] <imriz> they removed the configuration knob at some point
[8:44] <imriz> but not sure if it was backported to dumpling or not
[8:44] <Fruit> it should default to yes these days
[8:44] <imriz> even in dumpling?
[8:44] <imriz> btw, according to my quick sweep of the code
[8:44] <imriz> even when it is set to true
[8:45] <imriz> it still considers filestore_max_inline_xattr_size
[8:45] <imriz> so updates lower than filestore_max_inline_xattr_size still go to the FS?
[8:45] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[8:46] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[8:46] <imriz> it does "if (p->second.length() > g_conf->filestore_max_inline_xattr_size)" in FileStore.cc
[8:47] <imriz> so maybe filestore_max_inline_xattr_size should be adjusted too?
[8:47] <Fruit> imriz: not in dumpling afaik
[8:47] <Fruit> perhaps setting filestore_max_inline_xattr_size to 0 would help? dunno
[8:47] <imriz> well
[8:47] <imriz> I'll have to wait for people at Inktank to wake up :)
[8:48] * b0e (~aledermue@37-5-55-13-dynip.superkabel.de) Quit (Remote host closed the connection)
[8:49] <imriz> this is the scary part: http://tracker.ceph.com/issues/6143#note-8
[8:49] <imriz> "data corruption" and "here is no known fix for when the issue does occur" does not look very promising :)
[8:50] <imriz> The funny thing is that we downgraded from emperor to dumpling for the commercial support :)
[8:51] * haomaiwa_ (~haomaiwan@117.79.232.255) Quit (Ping timeout: 480 seconds)
[8:52] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) has joined #ceph
[8:53] * jtang1 (~jtang@80.111.79.253) has joined #ceph
[8:54] <imriz> Fruit, you don't have a dumpling cluster running by any chance, do you?
[8:55] <imriz> just wondering if "config show" should show the omap directive
[8:55] * kiwigeraint (~kiwigerai@208.72.139.54) Quit (Remote host closed the connection)
[8:56] <Fruit> I do
[8:56] <Fruit> but I'm running a more recent kernel
[8:57] * kiwigeraint (~kiwigerai@208.72.139.54) has joined #ceph
[8:57] * kiwigeraint (~kiwigerai@208.72.139.54) Quit (Remote host closed the connection)
[8:57] * kiwigeraint (~kiwigerai@208.72.139.54) has joined #ceph
[8:57] <imriz> but can you check if ceph --admin-daemon /var/run/ceph/ceph-osd.34.asok config show displays this setting?
[8:57] <imriz> please?
[8:57] <Fruit> sure hold on
[8:57] <imriz> thanks :)
[8:58] <Fruit> "filestore_debug_omap_check": "false",
[8:58] <imriz> same as I get
[8:58] <imriz> you have filestore_xattr_use_omap in your ceph.conf?
[8:59] <Fruit> filestore xattr use omap = true
[8:59] <imriz> so config show doesn't verify if it is on or off
[8:59] <imriz> fun
[8:59] <Fruit> http://sprunge.us/cjjd
[9:00] <Fruit> "osd_max_attr_size": "0",
[9:01] <imriz> mine is set to 0 too
[9:01] <Fruit> seems to be the default
[9:01] <Fruit> otoh:
[9:01] <Fruit> "filestore_max_inline_xattr_size": "512",
[9:01] <Fruit> "filestore_max_inline_xattrs": "2",
[9:01] <imriz> yeah
[9:01] <imriz> this is what I was about to say
[9:02] <imriz> that I think filestore_max_inline_xattr_size is the one that matters
[9:02] * Cube (~Cube@66.87.65.44) Quit (Ping timeout: 480 seconds)
[9:02] <Fruit> I'd set both of those to 0
[9:02] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[9:02] <Fruit> do you have scripts to reformat an osd?
[9:02] <imriz> yeah
[9:02] <imriz> fully puppetized
[9:02] <Fruit> nice
[9:03] <imriz> we got puppet to handle the cluster creation from 0 to 100
[9:03] <imriz> including radosgw
[9:03] <Fruit> we only use rbd atm
[9:04] <imriz> ah
[9:04] <imriz> for virtual machines?
[9:04] <Fruit> yep
[9:04] <imriz> xen? kvm?
[9:04] <Fruit> openstack kvm
[9:04] <imriz> awesome!
[9:05] <imriz> we want to go there too
[9:05] <imriz> you're happy with it?
[9:05] <Fruit> we're happy with ceph and kvm. openstack, hmm. it's nice, but it appears to be geared at a different problem set than ours
[9:05] <imriz> can you elaborate some more?
[9:06] <imriz> I would love to hear about it, since we are considering openstack now
[9:06] <Fruit> well openstack is designed to fire off like 100 throwaway vm's that you can destroy and re-create at will
[9:06] <Fruit> our applications aren't like that. they're precious snowflakes :P
[9:06] <imriz> :)
[9:07] <imriz> how does openstack achieve vm HA?
[9:07] <imriz> will it automatically restart VMs if an hypervisor fails?
[9:07] <Fruit> it doesn't, that's one of the problems
[9:07] <Fruit> "what do you mean HA? you have 90 remaining VM's, don't you?"
[9:07] <imriz> heh :)
[9:07] <imriz> you got me actually laughing :)
[9:08] <Fruit> :)
[9:08] <imriz> I am looking for a solution more like VMWare's HA
[9:08] <imriz> but without having to define each VM in pacemaker :)
[9:08] <Fruit> it does support live migration but you have to initiate it manually
[9:08] <Fruit> heh yeah, vmware refugees here as well.
[9:08] <Fruit> maybe I can script something
[9:09] <imriz> over openstack?
[9:09] * analbeard (~shw@support.memset.com) has joined #ceph
[9:09] <Fruit> yeah
[9:09] <Fruit> I do have to say the openstack<->ceph integration is very smooth so far
[9:09] <Fruit> no complaints there
[9:09] * haomaiwang (~haomaiwan@114.54.16.242) has joined #ceph
[9:09] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) has joined #ceph
[9:10] <Fruit> anyway, gotta run
[9:10] <Fruit> good luck with the attributes
[9:10] <imriz> thanks for the tips and help
[9:17] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[9:21] * Cube (~Cube@66-87-65-44.pools.spcsdns.net) has joined #ceph
[9:25] * MACscr (~Adium@c-50-158-183-38.hsd1.il.comcast.net) has joined #ceph
[9:25] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) has joined #ceph
[9:33] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[9:33] * BManojlovic (~steki@91.195.39.5) has joined #ceph
[9:33] * Sysadmin88 (~IceChat77@05452a50.skybroadband.com) Quit (Quit: Life without danger is a waste of oxygen)
[9:40] * Cataglottism (~Cataglott@dsl-087-195-030-184.solcon.nl) has joined #ceph
[9:41] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) has joined #ceph
[9:46] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[9:49] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[9:50] * TMM (~hp@c97185.upc-c.chello.nl) Quit (Quit: Ex-Chat)
[9:50] * analbeard (~shw@support.memset.com) Quit (Quit: Leaving.)
[9:52] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit (Quit: My MacBook has gone to sleep. ZZZzzz???)
[9:52] * Cube (~Cube@66-87-65-44.pools.spcsdns.net) Quit (Quit: Leaving.)
[9:54] * analbeard (~shw@support.memset.com) has joined #ceph
[9:57] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) has joined #ceph
[9:59] * jtang1 (~jtang@80.111.79.253) Quit (Quit: Leaving.)
[10:06] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[10:07] * jjgalvez (~jjgalvez@ip98-167-16-160.lv.lv.cox.net) Quit (Quit: Leaving.)
[10:08] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[10:11] * BManojlovic (~steki@91.195.39.5) Quit (Read error: Connection reset by peer)
[10:11] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit ()
[10:11] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[10:13] * Cube (~Cube@66-87-65-44.pools.spcsdns.net) has joined #ceph
[10:13] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) has joined #ceph
[10:13] * LeaChim (~LeaChim@host86-162-1-71.range86-162.btcentralplus.com) has joined #ceph
[10:18] * Cube (~Cube@66-87-65-44.pools.spcsdns.net) Quit ()
[10:18] * TMM (~hp@sams-office-nat.tomtomgroup.com) has joined #ceph
[10:21] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[10:23] * andreask (~andreask@h081217016175.dyn.cm.kabsi.at) has joined #ceph
[10:23] * ChanServ sets mode +v andreask
[10:29] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) has joined #ceph
[10:35] * thomnico (~thomnico@2a01:e35:8b41:120:a4c2:82ff:3e3d:224f) has joined #ceph
[10:37] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[10:38] * allsystemsarego (~allsystem@5-12-37-116.residential.rdsnet.ro) has joined #ceph
[10:41] * ofu (ofu@dedi3.fuckner.net) Quit (Ping timeout: 480 seconds)
[10:45] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) has joined #ceph
[10:46] * BManojlovic (~steki@178-221-115-97.dynamic.isp.telekom.rs) has joined #ceph
[10:46] * rdas (~rdas@nat-pool-pnq-t.redhat.com) Quit (Quit: Leaving)
[10:48] * Cube (~Cube@66-87-65-44.pools.spcsdns.net) has joined #ceph
[10:49] * rdas (~rdas@nat-pool-pnq-t.redhat.com) has joined #ceph
[10:51] * kumar (~shylesh@209.132.188.8) has joined #ceph
[10:51] * ofu (ofu@dedi3.fuckner.net) has joined #ceph
[10:53] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[11:00] * Cube (~Cube@66-87-65-44.pools.spcsdns.net) Quit (Ping timeout: 480 seconds)
[11:00] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) has joined #ceph
[11:01] * Cataglottism (~Cataglott@dsl-087-195-030-184.solcon.nl) Quit (Quit: My Mac Pro has gone to sleep. ZZZzzz???)
[11:07] * jcsp (~Adium@0001bf3a.user.oftc.net) has joined #ceph
[11:08] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[11:11] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[11:13] * gucki (~smuxi@p549F8F9B.dip0.t-ipconnect.de) has joined #ceph
[11:16] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) has joined #ceph
[11:18] * Cataglottism (~Cataglott@dsl-087-195-030-170.solcon.nl) has joined #ceph
[11:20] * oms101 (~oms101@charybdis-ext.suse.de) Quit (Quit: Leaving)
[11:24] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[11:24] * andrein__ (~andrein@46.108.33.138) Quit (Read error: Connection reset by peer)
[11:25] * zack_dolby (~textual@e0109-114-22-6-65.uqwimax.jp) Quit (Ping timeout: 480 seconds)
[11:25] * andrein__ (~andrein@46.108.33.138) has joined #ceph
[11:32] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) has joined #ceph
[11:32] * joao|lap (~JL@a95-92-33-54.cpe.netcabo.pt) has joined #ceph
[11:32] * ChanServ sets mode +o joao|lap
[11:34] * oms101 (~oms101@2620:113:80c0:5::2222) has joined #ceph
[11:36] * jbd_ (~jbd_@2001:41d0:52:a00::77) has joined #ceph
[11:39] * haomaiwang (~haomaiwan@114.54.16.242) Quit (Remote host closed the connection)
[11:39] * haomaiwang (~haomaiwan@113.196.160.69) has joined #ceph
[11:40] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[11:41] * vince (~vince@160.85.122.82) has joined #ceph
[11:43] * Cube (~Cube@66.87.65.44) has joined #ceph
[11:43] <vince> this page in the documentation http://ceph.com/docs/master/radosgw/adminops/ says: "An admin API request will be done on a URI that starts with the configurable ?admin? resource entry point.", but I cannot find the definition of this resource entry point elsewhere in the documentation. What is that supposed to be?
[11:44] * TMM (~hp@sams-office-nat.tomtomgroup.com) Quit (Quit: Ex-Chat)
[11:44] * TMM (~hp@sams-office-nat.tomtomgroup.com) has joined #ceph
[11:48] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) has joined #ceph
[11:51] * Cube (~Cube@66.87.65.44) Quit (Ping timeout: 480 seconds)
[11:52] * Cataglottism (~Cataglott@dsl-087-195-030-170.solcon.nl) Quit (Quit: My Mac Pro has gone to sleep. ZZZzzz???)
[11:53] * Cataglottism (~Cataglott@dsl-087-195-030-184.solcon.nl) has joined #ceph
[11:55] * haomaiwa_ (~haomaiwan@182.18.35.50) has joined #ceph
[11:56] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[11:59] * zack_dolby (~textual@e0109-114-22-6-65.uqwimax.jp) has joined #ceph
[12:00] * haomaiwang (~haomaiwan@113.196.160.69) Quit (Ping timeout: 480 seconds)
[12:04] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) has joined #ceph
[12:07] * zack_dolby (~textual@e0109-114-22-6-65.uqwimax.jp) Quit (Ping timeout: 480 seconds)
[12:07] * jjgalvez (~jjgalvez@ip98-167-16-160.lv.lv.cox.net) has joined #ceph
[12:10] * Cataglottism (~Cataglott@dsl-087-195-030-184.solcon.nl) Quit (Quit: My Mac Pro has gone to sleep. ZZZzzz???)
[12:12] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[12:18] * jjgalvez (~jjgalvez@ip98-167-16-160.lv.lv.cox.net) Quit (Quit: Leaving.)
[12:19] * jlogan (~Thunderbi@2600:c00:3010:1:1::40) Quit (Quit: jlogan)
[12:19] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) has joined #ceph
[12:20] * vince (~vince@160.85.122.82) Quit (Ping timeout: 480 seconds)
[12:21] * andreask (~andreask@h081217016175.dyn.cm.kabsi.at) Quit (Ping timeout: 480 seconds)
[12:24] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit (Quit: Leaving.)
[12:24] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[12:25] * thb (~me@0001bd58.user.oftc.net) Quit (Quit: Leaving.)
[12:26] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit ()
[12:28] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[12:31] * vince (~vince@160.85.231.155) has joined #ceph
[12:31] * AfC (~andrew@2407:7800:400:107f:2ad2:44ff:fe08:a4c) Quit (Ping timeout: 480 seconds)
[12:34] * Cataglottism (~Cataglott@dsl-087-195-030-184.solcon.nl) has joined #ceph
[12:35] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) has joined #ceph
[12:36] * mjeanson (~mjeanson@00012705.user.oftc.net) Quit (Quit: No Ping reply in 180 seconds.)
[12:37] * Cube (~Cube@66.87.65.44) has joined #ceph
[12:37] * rdas (~rdas@nat-pool-pnq-t.redhat.com) Quit (Ping timeout: 480 seconds)
[12:37] * mjeanson (~mjeanson@bell.multivax.ca) has joined #ceph
[12:39] * kiwigeraint (~kiwigerai@208.72.139.54) Quit (Remote host closed the connection)
[12:44] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[12:45] * Cube (~Cube@66.87.65.44) Quit (Ping timeout: 480 seconds)
[12:48] * topro (~prousa@host-62-245-142-50.customer.m-online.net) has joined #ceph
[12:49] * dosaboy (~dosaboy@65.93.189.91.lcy-01.canonistack.canonical.com) Quit (Quit: leaving)
[12:49] * dosaboy (~dosaboy@65.93.189.91.lcy-01.canonistack.canonical.com) has joined #ceph
[12:51] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) has joined #ceph
[12:53] * tracphil_ (~tracphil@pool-71-163-168-251.washdc.fios.verizon.net) Quit (Quit: leaving)
[12:55] * i_m (~ivan.miro@gbibp9ph1--blueice4n2.emea.ibm.com) has joined #ceph
[12:55] * i_m (~ivan.miro@gbibp9ph1--blueice4n2.emea.ibm.com) Quit ()
[12:56] * i_m (~ivan.miro@gbibp9ph1--blueice4n2.emea.ibm.com) has joined #ceph
[12:57] * i_m (~ivan.miro@gbibp9ph1--blueice4n2.emea.ibm.com) Quit ()
[12:58] * i_m (~ivan.miro@gbibp9ph1--blueice4n2.emea.ibm.com) has joined #ceph
[13:00] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[13:02] * andreask (~andreask@h081217016175.dyn.cm.kabsi.at) has joined #ceph
[13:02] * ChanServ sets mode +v andreask
[13:07] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) has joined #ceph
[13:13] * madkiss (~madkiss@ws14-185.sys11.net) has joined #ceph
[13:15] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[13:23] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) has joined #ceph
[13:28] * Cataglottism (~Cataglott@dsl-087-195-030-184.solcon.nl) Quit (Quit: My Mac Pro has gone to sleep. ZZZzzz???)
[13:29] * srenatus (~stephan@185.27.182.2) has joined #ceph
[13:31] <srenatus> hi there. I'm troubled by the following problem: having stored some objects in ceph via rgw+{swift,s3}+keystone, I'd like to be able to retrieve them without having keystone running...is there any way to do that? 'rados -p .rgw.buckets get default.XXX/file.data -o file.data' seems to be corrupted
[13:31] * Cube (~Cube@66.87.65.44) has joined #ceph
[13:31] * ckranz (~Chris@cpc26-salf5-2-0-cust77.10-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[13:31] <srenatus> s/-o//
[13:31] <kraken> srenatus meant to say: hi there. I'm troubled by the following problem: having stored some objects in ceph via rgw+{swift,s3}+keystone, I'd like to be able to retrieve them without having keystone running...is there any way to do that? 'rados -p .rgw.buckets get default.XXX/file.data file.data' seems to be corrupted
[13:32] <srenatus> oh hehe nice bot.
[13:32] <srenatus> hmm nevermind. tried a .py file and it works. my file.tar.enc in the other hand is not decryptable. weird.
[13:38] <srenatus> oh. the "rados"-retrieved file is only 512K big
[13:38] <srenatus> the .py file I tested was smaller of course
[13:39] * Cube (~Cube@66.87.65.44) Quit (Ping timeout: 480 seconds)
[13:40] * morse (~morse@supercomputing.univpm.it) has joined #ceph
[13:41] * vince (~vince@160.85.231.155) Quit (Quit: Leaving)
[14:06] * shang (~ShangWu@175.41.48.77) Quit (Remote host closed the connection)
[14:08] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[14:10] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit ()
[14:11] * diegows (~diegows@190.190.5.238) has joined #ceph
[14:14] * mtanski (~mtanski@cpe-72-229-51-156.nyc.res.rr.com) has joined #ceph
[14:17] <hybrid512> Hi
[14:17] <hybrid512> I have an OSD in a 3 nodes cluster that is trashed due to a failing hardware
[14:18] * thb (~me@2a02:2028:5b:1e80:6267:20ff:fec9:4e40) has joined #ceph
[14:18] <hybrid512> the drive is working but the XFS storage is corrupted
[14:18] <andreask> srenatus: hmm ... if it is a bigger file I'd say it consists of several objects and you need all of them to have the complete file
[14:18] * thb is now known as Guest5975
[14:18] <hybrid512> what do you recommend to restore that OSD ?
[14:19] <hybrid512> xfs_repair -n indicates error that can be repaired with flushing the log ... I think I'll get an unusable FS after that ...
[14:19] <hybrid512> should I do the repair and start my OSD again or should I waipe it completely ?
[14:19] <hybrid512> wipe
[14:19] <hybrid512> how ceph will react ?
[14:19] <hybrid512> please help :(
[14:20] <Serbitar> probably best to wipe it and add it again
[14:20] <Serbitar> if you say the FS is corrupt
[14:20] <Vacum> hybrid512: set the weight of the osd to 0.0, so the other OSDs can take over the pgs (->backfilling). once all data is backfilled and the osd is no more pgs, you can mark it "out". remove the osd from the cluster
[14:21] * Guest5975 is now known as thb
[14:21] <hybrid512> do I have to remove it from the CRUSH map ?
[14:21] <andreask> hybrid512: follow https://ceph.com/docs/master/rados/operations/add-or-rm-osds/ ... to remove the osd
[14:21] <Vacum> hybrid512: if all your pools have a replica size of >=2, you can of course just set it to "out" and the backfilling will happen from the remaining copy/copies
[14:21] <andreask> and then add it as new
[14:21] <Vacum> hybrid512: before adding that hdd again, run at least a smart test
[14:22] <Vacum> hybrid512: corruption is most likely due to a hw problem
[14:22] <hybrid512> my 2 other nodes already took over properly, "ceph health" gives me HEALTH_OK
[14:22] <hybrid512> my OSD is out already by now
[14:22] <Vacum> hybrid512: ok, then follow the link andreask gave you
[14:23] <Vacum> hybrid512: then, I suggest, before bringing in the hdd as a new osd, really test it intensively.
[14:23] * valeech (~valeech@166.170.29.50) has joined #ceph
[14:23] * mtanski (~mtanski@cpe-72-229-51-156.nyc.res.rr.com) Quit (Quit: mtanski)
[14:23] <hybrid512> in fact, my cluster is now running properly on 2 nodes, I'm just now with a failed OSD that can't start due to XFS corruption
[14:23] * wschulze (~wschulze@cpe-72-229-37-201.nyc.res.rr.com) has joined #ceph
[14:23] * pressureman (~pressurem@62.217.45.26) has joined #ceph
[14:23] <hybrid512> the HDD is not faluty, this is the RAID controler that failed
[14:24] <Vacum> hybrid512: one more reason to get rid of RAID controllers, no need for them with ceph ;)
[14:25] <hybrid512> I know but this is a rented dedicated server, I don't have much control on the hardware inside the beast
[14:25] * mtanski (~mtanski@cpe-72-229-51-156.nyc.res.rr.com) has joined #ceph
[14:25] <Vacum> hybrid512: wipe the whole drive, remove the partitions. are you using ceph-deploy? if yes, bring that hdd without any partition as a "fresh" drive back with ceph-deploy
[14:25] <Vacum> hybrid512: so the controller has already been changed to a new one?
[14:25] <hybrid512> no, I'm not using ceph-deploy, I created/added it manualy
[14:26] <hybrid512> so I have to remove it first following the procedure and add it again, right ?
[14:26] <hybrid512> that way it will be seen as new and the cluster will reweight automaticaly ?
[14:26] <Vacum> hybrid512: yes, remove it completely. do not add it again without removing all partitions and creating a new xfs fs
[14:27] * japuzzo (~japuzzo@pok2.bluebird.ibm.com) has joined #ceph
[14:27] <hybrid512> the controler hasn't been chaged, the cooling system was faulty, it got too hot and it simply crashed due to overheating
[14:27] <hybrid512> the cooling system has been changed
[14:27] <Vacum> ok
[14:27] <hybrid512> ok, I'll do that, thanks
[14:28] * rdas (~rdas@nat-pool-pnq-t.redhat.com) has joined #ceph
[14:31] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[14:31] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit ()
[14:32] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[14:33] * tracphil (~tracphil@130.14.71.217) has joined #ceph
[14:33] * zidarsk8 (~zidar@2001:1470:fffe:fe01:e2ca:94ff:fe34:7822) has joined #ceph
[14:33] * zidarsk8 (~zidar@2001:1470:fffe:fe01:e2ca:94ff:fe34:7822) has left #ceph
[14:33] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit ()
[14:34] <hybrid512> another question : as soon as my node failed, Ceph began a recovery on the remaining active nodes and it is quite ressource hungry ... simply filling my network bandwidth and my hard drives are working as hell ... is it normal ? is there a way to throttle it down so that it doesn't impact performances too much ?
[14:34] * shang (~ShangWu@175.41.48.77) has joined #ceph
[14:34] <hybrid512> I'm sorry but I'm still very new to Ceph, I'm more experimented with Sheepdog and I didn't had that kind of behaviour with Sheepdog :(
[14:35] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[14:36] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit ()
[14:37] * gucki (~smuxi@p549F8F9B.dip0.t-ipconnect.de) Quit (Read error: Connection reset by peer)
[14:38] <andreask> hybrid512: you can limit e.g. maximum backfill threads
[14:38] <andreask> osd max backfills
[14:39] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[14:40] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit ()
[14:42] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[14:43] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit ()
[14:44] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[14:46] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit ()
[14:48] * t0rn (~ssullivan@c-24-11-198-35.hsd1.mi.comcast.net) Quit (Remote host closed the connection)
[14:49] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[14:50] * sputnik1_ (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[14:50] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit (Read error: Connection reset by peer)
[14:50] * mtanski (~mtanski@cpe-72-229-51-156.nyc.res.rr.com) Quit (Quit: mtanski)
[14:51] * sputnik1_ (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit ()
[14:52] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[14:52] * sroy (~sroy@208.88.110.46) has joined #ceph
[14:52] * thuc (~thuc@c-71-198-202-49.hsd1.ca.comcast.net) has joined #ceph
[14:53] <hybrid512> thanks but I don't really understand how it works
[14:53] <hybrid512> I mean I don't understand which value is best
[14:53] <hybrid512> I see default is 10 but what does that mean ?
[14:54] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit ()
[14:54] <Fruit> hybrid512: you could test different values to see which one works best for your specific setup and hardware
[14:55] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[14:55] <hybrid512> Fruit: unfortunately, this cluster is already out of the test conditions and is used effectively so I'd prefer not playing too much with it unless necessary
[14:55] * KevinPerks (~Adium@cpe-066-026-252-218.triad.res.rr.com) has joined #ceph
[14:55] <Fruit> if it's too low the only drawback is that refills take a long time
[14:56] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit ()
[14:59] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[15:00] * sputnik1_ (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[15:00] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit (Read error: Connection reset by peer)
[15:00] * thuc (~thuc@c-71-198-202-49.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[15:00] <hybrid512> Fruit: thanks :)
[15:01] * analbeard (~shw@support.memset.com) Quit (Ping timeout: 480 seconds)
[15:01] * sputnik1_ (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit ()
[15:02] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[15:04] * analbeard (~shw@support.memset.com) has joined #ceph
[15:04] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit ()
[15:05] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[15:07] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit ()
[15:10] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[15:11] * sjm (~sjm@pool-108-53-56-179.nwrknj.fios.verizon.net) has joined #ceph
[15:12] * valeech (~valeech@166.170.29.50) Quit (Read error: No route to host)
[15:12] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit ()
[15:13] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[15:15] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit ()
[15:15] * valeech (~valeech@166.170.29.50) has joined #ceph
[15:16] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[15:17] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit ()
[15:17] * fghaas (~florian@91-119-229-245.dynamic.xdsl-line.inode.at) has joined #ceph
[15:18] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[15:19] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit ()
[15:20] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[15:21] * mtanski (~mtanski@cpe-72-229-51-156.nyc.res.rr.com) has joined #ceph
[15:22] * mtanski (~mtanski@cpe-72-229-51-156.nyc.res.rr.com) Quit ()
[15:23] * BillK (~BillK-OFT@106-68-27-227.dyn.iinet.net.au) Quit (Ping timeout: 480 seconds)
[15:24] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit ()
[15:25] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[15:27] * timidshark (~timidshar@74.118.238.209) has joined #ceph
[15:27] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit ()
[15:28] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[15:29] * dock (~dock@h-199-142.a137.corp.bahnhof.se) has joined #ceph
[15:30] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit ()
[15:31] * kiwigeraint (~kiwigerai@208.72.139.54) has joined #ceph
[15:31] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[15:31] <dock> Hi, after moving around between OSDs today we had a major incident, we followed the manual when removing OSDs and it resulted in massive unavailability
[15:32] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit ()
[15:33] <dock> When we did the "ceph osd crush remove {name}" step all nodes were reweighted, that made almost all the PGs to get rebalanced, note that the cluster was at this time already perfectly balanced
[15:33] * jeff-YF (~jeffyf@67.23.123.228) has joined #ceph
[15:34] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[15:34] <dock> We are guessing that the PG-hash was updated as it should rely on the weight, when we manually changed the weights back, the cluster agreed that it was infact in balance.
[15:35] * zack_dolby (~textual@p852cae.tokynt01.ap.so-net.ne.jp) has joined #ceph
[15:35] <hybrid512> thanks everyone, my cluster is recovering perfectly
[15:36] <dock> The unavailability this caused was that mapping ceased to work, in some cases it worked but when we tried writing we got EIO
[15:36] <dock> This last part should never have happened
[15:36] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit ()
[15:36] * kumar (~shylesh@209.132.188.8) Quit (Read error: Operation timed out)
[15:37] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[15:38] <dock> How can we make the rbd block devices block in situations instead for causing syscalls to fail with EIO?
[15:38] * alfredodeza (~alfredode@198.206.133.89) has left #ceph
[15:38] <dock> in situations like this(
[15:39] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit ()
[15:39] <dock> We did see errors like Apr 09 13:14:37 b0-host-338 kernel: rbd: rbd27: result -6 xferred 1000
[15:39] <dock> Apr 09 13:14:37 b0-host-338 kernel: end_request: I/O error, dev rbd27, sector 70920
[15:40] * thuc (~thuc@c-71-198-202-49.hsd1.ca.comcast.net) has joined #ceph
[15:40] <dock> It is unacceptable that we get errnos just because the cluster is rebalancing
[15:40] <kraken> http://i.imgur.com/BwdP2xl.gif
[15:41] * kiwigeraint (~kiwigerai@208.72.139.54) Quit (Remote host closed the connection)
[15:42] * valeech (~valeech@166.170.29.50) Quit (Quit: valeech)
[15:42] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[15:43] * sputnik1_ (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[15:43] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit (Read error: Connection reset by peer)
[15:44] * jeff-YF (~jeffyf@67.23.123.228) Quit (Quit: jeff-YF)
[15:44] * sputnik1_ (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit ()
[15:45] <dock> Because if ceph doesn't guarantee that, there is effectively no availability guarantees
[15:45] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[15:47] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit ()
[15:48] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[15:49] * thuc (~thuc@c-71-198-202-49.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[15:49] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit ()
[15:50] * jtangwk (~Adium@gateway.tchpc.tcd.ie) Quit (Quit: Leaving.)
[15:50] <srenatus> andreask: indeed. problem is, I dont know how
[15:50] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[15:51] <Gugge-47527> dock: that sounds like a bug, are you using a faily new kernel on the rbd client?
[15:52] <dock> Yes it is very new
[15:52] <dock> I believe it is 3.14
[15:52] <Gugge-47527> should be new enough
[15:52] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit ()
[15:53] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[15:55] <dock> We are digging through the rbd client source, see if we are missing any fixes
[16:07] * markbby (~Adium@168.94.245.3) has joined #ceph
[16:08] * analbeard1 (~shw@support.memset.com) has joined #ceph
[16:09] * analbeard2 (~shw@support.memset.com) has joined #ceph
[16:11] * analbeard3 (~shw@188.29.165.63.threembb.co.uk) has joined #ceph
[16:11] * analbeard1 (~shw@support.memset.com) Quit (Read error: No route to host)
[16:11] * analbeard (~shw@support.memset.com) Quit (Read error: No route to host)
[16:12] * Cube (~Cube@66-87-65-44.pools.spcsdns.net) has joined #ceph
[16:15] * Cube (~Cube@66-87-65-44.pools.spcsdns.net) Quit ()
[16:16] * ircolle (~Adium@c-67-172-132-222.hsd1.co.comcast.net) has joined #ceph
[16:17] * analbeard2 (~shw@support.memset.com) Quit (Ping timeout: 480 seconds)
[16:23] * jeff-YF (~jeffyf@67.23.117.122) has joined #ceph
[16:23] * mtanski (~mtanski@69.193.178.202) has joined #ceph
[16:23] * jksM (~jks@3e6b5724.rev.stofanet.dk) Quit (Ping timeout: 480 seconds)
[16:23] <dock> Not that many rbd patches to go through and I didn't see anything that describes my problems, so I assume it is an unfixed bug if it is a bug. We will eventually try to reproduce it but I don't see it happening today
[16:23] <dock> Or this week
[16:23] * analbeard3 (~shw@188.29.165.63.threembb.co.uk) Quit (Ping timeout: 480 seconds)
[16:25] * xarses (~andreww@c-24-23-183-44.hsd1.ca.comcast.net) has joined #ceph
[16:27] * joef (~Adium@2620:79:0:131:3127:3cb2:331f:d320) Quit (Remote host closed the connection)
[16:29] <fghaas> are you getting the same errors if you're reading rbd from userspace, say with rbd export?
[16:30] * andrein__ (~andrein@46.108.33.138) Quit (Read error: No route to host)
[16:31] * andrein__ (~andrein@46.108.33.138) has joined #ceph
[16:31] <fghaas> and are you also sure that your "major incident" didn't leave you with unfound objects or stuck PGs?
[16:31] * zack_dolby (~textual@p852cae.tokynt01.ap.so-net.ne.jp) Quit (Read error: Connection reset by peer)
[16:31] * zack_dolby (~textual@p852cae.tokynt01.ap.so-net.ne.jp) has joined #ceph
[16:32] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit (Quit: My MacBook has gone to sleep. ZZZzzz???)
[16:34] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[16:35] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit ()
[16:35] * rotbeard (~redbeard@80.150.49.50) has joined #ceph
[16:37] * jcsp1 (~Adium@82-71-55-202.dsl.in-addr.zen.co.uk) has joined #ceph
[16:37] * glambert (~glambert@37.157.50.80) has joined #ceph
[16:38] * analbeard (~shw@support.memset.com) has joined #ceph
[16:40] * andreask (~andreask@h081217016175.dyn.cm.kabsi.at) Quit (Ping timeout: 480 seconds)
[16:41] * Cataglottism (~Cataglott@dsl-087-195-030-170.solcon.nl) has joined #ceph
[16:41] <dock> fghaas: The major incident i refered to was that rebalancing caused failures in userspace, the pool returned to healthy almost instantly after we restored the altered weights
[16:41] * jcsp (~Adium@0001bf3a.user.oftc.net) Quit (Ping timeout: 480 seconds)
[16:42] <fghaas> which ceph version are you on?
[16:42] <dock> I won't reproduce it in production, so I will have to set up another pool before I start properly debug this. Most likely not this week sadly
[16:47] <dock> emperor
[16:48] * vata (~vata@2607:fad8:4:6:4467:6b20:2e09:8f35) has joined #ceph
[16:48] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[16:51] * thomnico (~thomnico@2a01:e35:8b41:120:a4c2:82ff:3e3d:224f) Quit (Quit: Ex-Chat)
[16:56] <med> KevinPerks what's sheldon's IRC nickname (I don't recall)
[16:56] * gregmark (~Adium@68.87.42.115) has joined #ceph
[16:57] * thomnico (~thomnico@2a01:e35:8b41:120:a4c2:82ff:3e3d:224f) has joined #ceph
[16:57] * rdas (~rdas@nat-pool-pnq-t.redhat.com) Quit (Quit: Leaving)
[16:58] * hasues (~hazuez@108-236-232-243.lightspeed.knvltn.sbcglobal.net) Quit (Quit: Leaving.)
[16:58] <ircolle> med - sheldon is sjm
[16:59] <sjm> I am here
[16:59] <med> ircolle, sjm, thanks
[16:59] * rread (~rread@205.201.183.2) has joined #ceph
[17:01] <KevinPerks> hi dave
[17:03] * analbeard (~shw@support.memset.com) Quit (Quit: Leaving.)
[17:04] * rdas (~rdas@nat-pool-pnq-t.redhat.com) has joined #ceph
[17:06] * rread (~rread@205.201.183.2) Quit (Quit: rread)
[17:08] * joao|lap (~JL@a95-92-33-54.cpe.netcabo.pt) Quit (Ping timeout: 480 seconds)
[17:09] * thuc (~thuc@c-71-198-202-49.hsd1.ca.comcast.net) has joined #ceph
[17:10] * rotbeard (~redbeard@80.150.49.50) Quit (Ping timeout: 480 seconds)
[17:10] * rread (~rread@205.201.183.2) has joined #ceph
[17:10] * rread (~rread@205.201.183.2) has left #ceph
[17:17] * thuc (~thuc@c-71-198-202-49.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[17:17] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit (Quit: My MacBook has gone to sleep. ZZZzzz???)
[17:18] * cfreak200 (~cfreak200@p4FF3EAE4.dip0.t-ipconnect.de) has joined #ceph
[17:19] * jpm (~jpm@kwak.nosuchthing.org) has joined #ceph
[17:24] * Georgyo (~georgyo@shamm.as) has joined #ceph
[17:28] * andreask (~andreask@h081217016175.dyn.cm.kabsi.at) has joined #ceph
[17:28] * ChanServ sets mode +v andreask
[17:28] * mattch (~mattch@pcw3047.see.ed.ac.uk) Quit (Remote host closed the connection)
[17:29] * mattch (~mattch@pcw3047.see.ed.ac.uk) has joined #ceph
[17:29] * bitblt (~don@128-107-239-233.cisco.com) has joined #ceph
[17:30] * xarses (~andreww@c-24-23-183-44.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[17:30] * JeffK (~Jeff@38.99.52.10) has joined #ceph
[17:31] * madkiss (~madkiss@ws14-185.sys11.net) Quit (Quit: Leaving.)
[17:32] * thomnico (~thomnico@2a01:e35:8b41:120:a4c2:82ff:3e3d:224f) Quit (Quit: Ex-Chat)
[17:33] * yuriw1 (~Adium@c-71-202-126-141.hsd1.ca.comcast.net) has joined #ceph
[17:33] * yuriw (~Adium@c-71-202-126-141.hsd1.ca.comcast.net) Quit (Read error: Connection reset by peer)
[17:35] * Cataglottism (~Cataglott@dsl-087-195-030-170.solcon.nl) Quit (Quit: My Mac Pro has gone to sleep. ZZZzzz???)
[17:44] * BManojlovic (~steki@178-221-115-97.dynamic.isp.telekom.rs) Quit (Quit: Ja odoh a vi sta 'ocete...)
[17:46] * dock (~dock@h-199-142.a137.corp.bahnhof.se) Quit (Quit: Lost terminal)
[17:47] * jlogan (~Thunderbi@2600:c00:3010:1:1::40) has joined #ceph
[17:52] * rmoe (~quassel@173-228-89-134.dsl.static.sonic.net) Quit (Ping timeout: 480 seconds)
[17:54] * gregsfortytwo1 (~Adium@cpe-172-250-69-138.socal.res.rr.com) has joined #ceph
[17:54] * alram (~alram@cpe-76-167-50-51.socal.res.rr.com) has joined #ceph
[17:56] * thb (~me@0001bd58.user.oftc.net) Quit (Ping timeout: 480 seconds)
[17:57] * JoeGruher (~JoeGruher@jfdmzpr04-ext.jf.intel.com) has joined #ceph
[17:58] * andreask (~andreask@h081217016175.dyn.cm.kabsi.at) has left #ceph
[17:59] * srenatus (~stephan@185.27.182.2) Quit (Quit: bye)
[18:01] * mattch (~mattch@pcw3047.see.ed.ac.uk) Quit (Remote host closed the connection)
[18:01] * mattch (~mattch@pcw3047.see.ed.ac.uk) has joined #ceph
[18:05] * fghaas (~florian@91-119-229-245.dynamic.xdsl-line.inode.at) Quit (Quit: Leaving.)
[18:06] * Georgyo (~georgyo@shamm.as) Quit (Remote host closed the connection)
[18:06] * Georgyo (~georgyo@shamm.as) has joined #ceph
[18:09] * xarses (~andreww@12.164.168.117) has joined #ceph
[18:10] * rmoe (~quassel@12.164.168.117) has joined #ceph
[18:12] * timidshark (~timidshar@74.118.238.209) Quit (Remote host closed the connection)
[18:12] <singler> I have locked/hang ceph-fuse (cephfs, centos 6.5, 0.72.2-0.el6). It uses 100% cpu, strace shows "futex(0x2d3dd20, FUTEX_WAIT_PRIVATE, 2, NULL"
[18:13] <singler> how could I debug it further? I do not see any logs
[18:14] * hasues (~hasues@kwfw01.scrippsnetworksinteractive.com) has joined #ceph
[18:23] * i_m (~ivan.miro@gbibp9ph1--blueice4n2.emea.ibm.com) Quit (Quit: Leaving.)
[18:24] * boichev (~boichev@213.169.56.130) has joined #ceph
[18:25] <boichev> can someone tell me how to proceed ? ceph healt hives me HEALTH_WARN 43 pgs degraded; 196 pgs stale; 196 pgs stuck stale; 196 pgs stuck unclean; recovery 7/28 objects degraded (25.000%) while I set the size of all pools to 1
[18:29] * sputnik13 (~sputnik13@207.8.121.241) has joined #ceph
[18:30] * TMM (~hp@sams-office-nat.tomtomgroup.com) Quit (Quit: Ex-Chat)
[18:37] * pressureman (~pressurem@62.217.45.26) Quit (Quit: Ex-Chat)
[18:40] * rdas (~rdas@nat-pool-pnq-t.redhat.com) Quit (Quit: Leaving)
[18:40] * kiwigeraint (~kiwigerai@208.72.139.54) has joined #ceph
[18:41] * nrs_ (~nrs@64.237.37.116) has joined #ceph
[18:42] <med> hey KevinPerks sjm: thanks for the call.
[18:44] <KevinPerks> np
[18:45] * zerick (~eocrospom@190.187.21.53) has joined #ceph
[18:47] * mtanski (~mtanski@69.193.178.202) Quit (Quit: mtanski)
[18:50] * fghaas (~florian@91-119-229-245.dynamic.xdsl-line.inode.at) has joined #ceph
[18:51] * BManojlovic (~steki@cable-94-189-160-252.dynamic.sbb.rs) has joined #ceph
[18:55] * mtanski (~mtanski@69.193.178.202) has joined #ceph
[18:56] <nrs_> guys...i heard that Ceph really wants the cluster network to be lossless...
[18:57] <nrs_> since all the replication is via TCP, doesn't the congestion avoidance mechanism of TCP take care of lost packets?
[18:58] * TMM (~hp@c97185.upc-c.chello.nl) has joined #ceph
[18:59] * kiwigeraint (~kiwigerai@208.72.139.54) Quit (Remote host closed the connection)
[18:59] * fghaas (~florian@91-119-229-245.dynamic.xdsl-line.inode.at) Quit (Quit: Leaving.)
[19:00] * jbd_ (~jbd_@2001:41d0:52:a00::77) has left #ceph
[19:00] <xarses> nrs_: I've not heard that, but replaying packets could seriously reduce performance
[19:01] <xarses> nrs_: any wired network with more 0.5% loss is an issue in my book
[19:01] <nrs_> well...loss in an IP network comes from congestion mostly
[19:01] * xdeller (~xdeller@109.188.124.66) Quit (Ping timeout: 480 seconds)
[19:02] * dpippenger (~riven@cpe-198-72-157-189.socal.res.rr.com) has joined #ceph
[19:02] <nrs_> what i'd like to know is if there are any Ceph users out there making use of any of the lossless Ethernet DCB technologies
[19:02] <nrs_> ETS, PFC, DCB-X, etc
[19:02] <xarses> you probably should write a message to the mail list
[19:03] <nrs_> i may
[19:11] * timidshark (~timidshar@70-88-62-73-fl.naples.hfc.comcastbusiness.net) has joined #ceph
[19:13] * nrs_ (~nrs@64.237.37.116) Quit (Quit: My MacBook has gone to sleep. ZZZzzz???)
[19:13] * nrs_ (~nrs@64.237.37.116) has joined #ceph
[19:14] * Cube (~Cube@66-87-65-44.pools.spcsdns.net) has joined #ceph
[19:16] * bjornar (~bjornar@ti0099a430-0023.bb.online.no) has joined #ceph
[19:16] <bjornar> Is it planned to support bare-metal (pxe) booting from rbd?
[19:17] <Fruit> you mean pulling the kernel and initrd from an rbd?
[19:18] * Qu310 (~Qu310@ip-121-0-1-110.static.dsl.onqcomms.net) Quit ()
[19:19] * Cataglottism (~Cataglott@dsl-087-195-030-184.solcon.nl) has joined #ceph
[19:21] * sjustwork (~sam@2607:f298:a:607:5a8:7c53:a54c:c1ff) has joined #ceph
[19:21] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[19:21] * kiwigeraint (~kiwigerai@208.72.139.54) has joined #ceph
[19:23] <bjornar> no..
[19:23] * kiwigeraint (~kiwigerai@208.72.139.54) Quit (Read error: Connection reset by peer)
[19:23] <bjornar> thats true.. could ofcorse serve kernel and initrd from pxe and set / to rbd
[19:23] * kiwigeraint (~kiwigerai@208.72.139.54) has joined #ceph
[19:25] * nrs_ (~nrs@64.237.37.116) Quit (Quit: My MacBook has gone to sleep. ZZZzzz???)
[19:25] <xarses> bjornar: works already in openstack with kvm. I'd guess that if you load a real initrd from pxe, you could get root from rbd on a newer kernel
[19:28] * rmoe (~quassel@12.164.168.117) Quit (Read error: Operation timed out)
[19:29] <Fruit> it doesn't sound like a very good idea. qemu's rbd implementation is much more robust than the kernel's
[19:30] * kiwigeraint (~kiwigerai@208.72.139.54) Quit (Remote host closed the connection)
[19:31] * asadpanda (~asadpanda@67.231.236.80) Quit (Quit: ZNC - http://znc.in)
[19:32] * rmoe (~quassel@12.164.168.117) has joined #ceph
[19:33] * thb (~me@port-21482.pppoe.wtnet.de) has joined #ceph
[19:34] * thb is now known as Guest5999
[19:37] * Guest5999 is now known as thb
[19:37] * Cataglottism (~Cataglott@dsl-087-195-030-184.solcon.nl) Quit (Quit: My Mac Pro has gone to sleep. ZZZzzz???)
[19:40] * kiwigeraint (~kiwigerai@208.72.139.54) has joined #ceph
[19:44] * imriz (~imriz@5.29.200.177) Quit (Ping timeout: 480 seconds)
[19:47] * Boltsky (~textual@cpe-198-72-138-106.socal.res.rr.com) Quit (Quit: My MacBook Pro has gone to sleep. ZZZzzz???)
[19:54] * asadpanda (~asadpanda@67.231.236.80) has joined #ceph
[20:00] <bjornar> Fruit, isnt the kernel coming along?
[20:01] <Fruit> it will probably always lag behind
[20:02] * fghaas (~florian@91-119-229-245.dynamic.xdsl-line.inode.at) has joined #ceph
[20:04] * mjblw1 (~mbaysek@wsip-174-79-34-244.ph.ph.cox.net) Quit (Remote host closed the connection)
[20:04] <bjornar> always is a very long time
[20:05] <bjornar> Fruit, I dont know you, are you speaking as an experienced developer, or a user with strong opinions?
[20:05] * mtanski (~mtanski@69.193.178.202) Quit (Quit: mtanski)
[20:05] <Fruit> bjornar: definitely the latter :)
[20:07] <Fruit> I'm only in this channel since a week or so, but I've seen several people complain about kernel rdb and none about qemu.
[20:07] <Fruit> (yes I know, the plural of anecdote is not data, etc, etc)
[20:08] * jbd (~jbd@2001:41d0:52:a00::77) has joined #ceph
[20:11] <ircolle> Fruit - user space rbd is more feature-full than kernel rbd
[20:12] * jbd (~jbd@2001:41d0:52:a00::77) has left #ceph
[20:13] * jbd (~jbd@2001:41d0:52:a00::77) has joined #ceph
[20:17] * jbd (~jbd@2001:41d0:52:a00::77) has left #ceph
[20:18] * jbd (~jbd@2001:41d0:52:a00::77) has joined #ceph
[20:19] * mtanski (~mtanski@69.193.178.202) has joined #ceph
[20:20] * JoeGruher (~JoeGruher@jfdmzpr04-ext.jf.intel.com) Quit (Remote host closed the connection)
[20:22] * odi (~quassel@2a00:12c0:1015:136::9) has joined #ceph
[20:23] * xarses (~andreww@12.164.168.117) Quit (Ping timeout: 480 seconds)
[20:24] * jtang1 (~jtang@80.111.79.253) has joined #ceph
[20:24] * JoeGruher (~JoeGruher@jfdmzpr04-ext.jf.intel.com) has joined #ceph
[20:24] * imriz (~imriz@5.29.200.177) has joined #ceph
[20:25] * Boltsky (~textual@64.191.249.70) has joined #ceph
[20:28] * jbd (~jbd@2001:41d0:52:a00::77) has left #ceph
[20:28] * jbd (~jbd@2001:41d0:52:a00::77) has joined #ceph
[20:30] * BManojlovic (~steki@cable-94-189-160-252.dynamic.sbb.rs) Quit (Quit: Ja odoh a vi sta 'ocete...)
[20:33] * odi (~quassel@2a00:12c0:1015:136::9) Quit (Remote host closed the connection)
[20:33] * kiwigeraint (~kiwigerai@208.72.139.54) Quit (Remote host closed the connection)
[20:34] * mtanski (~mtanski@69.193.178.202) Quit (Quit: mtanski)
[20:34] * xarses (~andreww@12.164.168.117) has joined #ceph
[20:35] * mtanski (~mtanski@69.193.178.202) has joined #ceph
[20:36] * odi (~quassel@2a00:12c0:1015:136::9) has joined #ceph
[20:38] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit (Quit: Leaving.)
[20:38] * jbd (~jbd@2001:41d0:52:a00::77) has left #ceph
[20:39] * kiwigeraint (~kiwigerai@208.72.139.54) has joined #ceph
[20:40] * Boltsky (~textual@64.191.249.70) Quit (Quit: My MacBook Pro has gone to sleep. ZZZzzz???)
[20:42] * jbd (~jbd@2001:41d0:52:a00::77) has joined #ceph
[20:44] * Boltsky (~textual@64.191.249.70) has joined #ceph
[20:45] * ChrisNBlum (~Adium@dhcp-ip-230.dorf.rwth-aachen.de) has joined #ceph
[20:45] * ChrisNBlum (~Adium@dhcp-ip-230.dorf.rwth-aachen.de) has left #ceph
[20:45] * alfredodeza (~alfredode@198.206.133.89) has joined #ceph
[20:47] * redondos (~redondos@204-16-155-82-static.ipnetworksinc.net) has joined #ceph
[20:51] * danieagle (~Daniel@186.214.56.177) has joined #ceph
[20:53] * kiwigeraint (~kiwigerai@208.72.139.54) Quit (Remote host closed the connection)
[20:58] <redondos> mmgaggle: are the ceph-cookbooks tested with dumpling on ubuntu precise?
[20:58] <redondos> I meant with emperor
[21:02] * Boltsky (~textual@64.191.249.70) Quit (Quit: My MacBook Pro has gone to sleep. ZZZzzz???)
[21:03] * koleosfuscus (~koleosfus@adsl-84-226-68-69.adslplus.ch) has joined #ceph
[21:06] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[21:07] * xarses (~andreww@12.164.168.117) Quit (Ping timeout: 480 seconds)
[21:07] * Georgyo (~georgyo@shamm.as) Quit (Quit: http://quassel-irc.org - Chat comfortably. Anywhere.)
[21:08] * Georgyo (~georgyo@shamm.as) has joined #ceph
[21:08] * jcsp (~Adium@82-71-55-202.dsl.in-addr.zen.co.uk) has joined #ceph
[21:08] * jcsp1 (~Adium@82-71-55-202.dsl.in-addr.zen.co.uk) Quit (Read error: Connection reset by peer)
[21:10] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[21:11] * jbd (~jbd@2001:41d0:52:a00::77) has left #ceph
[21:15] * jbd (~jbd@2001:41d0:52:a00::77) has joined #ceph
[21:15] * xarses (~andreww@12.164.168.117) has joined #ceph
[21:17] * Boltsky (~textual@64.191.249.70) has joined #ceph
[21:19] * jbd (~jbd@2001:41d0:52:a00::77) has left #ceph
[21:19] * jbd (~jbd@2001:41d0:52:a00::77) has joined #ceph
[21:20] * allsystemsarego (~allsystem@5-12-37-116.residential.rdsnet.ro) Quit (Quit: Leaving)
[21:25] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[21:25] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[21:30] * BManojlovic (~steki@cable-94-189-160-252.dynamic.sbb.rs) has joined #ceph
[21:33] * kiwigeraint (~kiwigerai@208.72.139.54) has joined #ceph
[21:33] * muratt (~medineli@88.234.96.23) has joined #ceph
[21:34] * muratt (~medineli@88.234.96.23) Quit (autokilled: Do not spam other people. Mail support@oftc.net if you feel this is in error. (2014-04-09 19:34:14))
[21:45] * Boltsky (~textual@64.191.249.70) Quit (Quit: My MacBook Pro has gone to sleep. ZZZzzz???)
[21:49] * jbd (~jbd@2001:41d0:52:a00::77) has left #ceph
[21:50] * jbd (~jbd@2001:41d0:52:a00::77) has joined #ceph
[21:56] * Pedras (~Adium@216.207.42.132) has joined #ceph
[21:57] * kiwigeraint (~kiwigerai@208.72.139.54) Quit (Remote host closed the connection)
[21:58] * JoeGruher (~JoeGruher@jfdmzpr04-ext.jf.intel.com) Quit (Remote host closed the connection)
[22:04] * sapna (~shiva@106.220.34.231) has joined #ceph
[22:04] * jbd (~jbd@2001:41d0:52:a00::77) has left #ceph
[22:05] * sapna (~shiva@106.220.34.231) Quit (Quit: Leaving)
[22:05] * jbd (~jbd@2001:41d0:52:a00::77) has joined #ceph
[22:08] * kiwigeraint (~kiwigerai@208.72.139.54) has joined #ceph
[22:10] * markbby (~Adium@168.94.245.3) Quit (Quit: Leaving.)
[22:10] * markbby (~Adium@168.94.245.3) has joined #ceph
[22:12] * tracphil (~tracphil@130.14.71.217) Quit (Quit: leaving)
[22:12] * fghaas (~florian@91-119-229-245.dynamic.xdsl-line.inode.at) Quit (Quit: Leaving.)
[22:15] * madkiss (~madkiss@37-5-55-31-dynip.superkabel.de) has joined #ceph
[22:17] * garphy`aw is now known as garphy
[22:20] * madkiss (~madkiss@37-5-55-31-dynip.superkabel.de) Quit (Quit: Leaving.)
[22:22] * sjm (~sjm@pool-108-53-56-179.nwrknj.fios.verizon.net) Quit (Ping timeout: 480 seconds)
[22:25] * sroy (~sroy@208.88.110.46) Quit (Quit: Quitte)
[22:27] <baffle> Has anyone looked a getting RGW to support Keystone/Identity V3 API and domains?
[22:27] <baffle> I persume it doesn't right now. :)
[22:34] * Sysadmin88 (~IceChat77@05452a50.skybroadband.com) has joined #ceph
[22:39] * andreask (~andreask@h081217016175.dyn.cm.kabsi.at) has joined #ceph
[22:39] * ChanServ sets mode +v andreask
[22:41] * jbd (~jbd@2001:41d0:52:a00::77) has left #ceph
[22:41] * sjm (~sjm@172.56.36.135) has joined #ceph
[22:41] * imriz (~imriz@5.29.200.177) Quit (Ping timeout: 480 seconds)
[22:46] * The_Bishop (~bishop@77.87.49.47) has joined #ceph
[22:48] * Boltsky (~textual@64.191.249.70) has joined #ceph
[22:53] * JoeGruher (~JoeGruher@jfdmzpr04-ext.jf.intel.com) has joined #ceph
[22:54] * Boltsky (~textual@64.191.249.70) Quit (Quit: My MacBook Pro has gone to sleep. ZZZzzz???)
[22:58] * timidshark (~timidshar@70-88-62-73-fl.naples.hfc.comcastbusiness.net) Quit (Quit: Leaving...)
[23:02] * jtang1 (~jtang@80.111.79.253) Quit (Quit: Leaving.)
[23:02] * mtanski (~mtanski@69.193.178.202) Quit (Quit: mtanski)
[23:03] * markbby (~Adium@168.94.245.3) Quit (Quit: Leaving.)
[23:06] * diegows (~diegows@190.190.5.238) Quit (Read error: Operation timed out)
[23:06] * markbby (~Adium@168.94.245.3) has joined #ceph
[23:12] * mtanski (~mtanski@69.193.178.202) has joined #ceph
[23:14] * The_Bishop (~bishop@77.87.49.47) Quit (Ping timeout: 480 seconds)
[23:29] * vata (~vata@2607:fad8:4:6:4467:6b20:2e09:8f35) Quit (Quit: Leaving.)
[23:29] <xarses> baffle: you should poke yehudasa on #ceph-devel, if my memory is correct they are the primary for RGW/Keystone
[23:30] * diegows (~diegows@190.190.5.238) has joined #ceph
[23:32] * jtang1 (~jtang@80.111.79.253) has joined #ceph
[23:34] * bjornar (~bjornar@ti0099a430-0023.bb.online.no) Quit (Ping timeout: 480 seconds)
[23:35] * gregmark (~Adium@68.87.42.115) Quit (Quit: Leaving.)
[23:36] * sjm (~sjm@172.56.36.135) Quit (Ping timeout: 480 seconds)
[23:38] * koleosfuscus (~koleosfus@adsl-84-226-68-69.adslplus.ch) has left #ceph
[23:40] * Boltsky (~textual@64.191.249.70) has joined #ceph
[23:40] * jtang1 (~jtang@80.111.79.253) Quit (Ping timeout: 480 seconds)
[23:42] * kiwigeraint (~kiwigerai@208.72.139.54) Quit (Remote host closed the connection)
[23:43] * The_Bishop (~bishop@2001:470:50b6:0:70e3:bb97:c2b8:98d4) has joined #ceph
[23:43] * mtanski (~mtanski@69.193.178.202) Quit (Ping timeout: 480 seconds)
[23:44] * jdmason (~jon@134.134.137.71) Quit (Read error: Connection reset by peer)
[23:45] * jdmason (~jon@192.55.54.38) has joined #ceph
[23:47] * nhm_ (~nhm@184-97-144-70.mpls.qwest.net) Quit (Ping timeout: 480 seconds)
[23:48] * reed (~reed@net-2-40-207-173.cust.dsl.teletu.it) Quit (Ping timeout: 480 seconds)
[23:48] * japuzzo (~japuzzo@pok2.bluebird.ibm.com) Quit (Quit: Leaving)
[23:54] * JoeGruher (~JoeGruher@jfdmzpr04-ext.jf.intel.com) Quit (Remote host closed the connection)
[23:55] * Boltsky (~textual@64.191.249.70) Quit (Quit: My MacBook Pro has gone to sleep. ZZZzzz???)
[23:56] * BillK (~BillK-OFT@106-68-27-227.dyn.iinet.net.au) has joined #ceph
[23:59] <baffle> xarses: Thanks. :)

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.