#ceph IRC Log

Index

IRC Log for 2015-08-03

Timestamps are in GMT/BST.

[0:09] * cholcombe (~chris@c-73-180-29-35.hsd1.or.comcast.net) has joined #ceph
[0:10] * Plesioth (~geegeegee@7R2AADCPB.tor-irc.dnsbl.oftc.net) Quit ()
[0:11] * darkid (~Diablothe@tor-exit0-readme.dfri.se) has joined #ceph
[0:13] * primechuck (~primechuc@173-17-128-216.client.mchsi.com) Quit (Remote host closed the connection)
[0:15] * i_m (~ivan.miro@88.206.99.242) has joined #ceph
[0:23] * primechuck (~primechuc@173-17-128-216.client.mchsi.com) has joined #ceph
[0:33] * primechuck (~primechuc@173-17-128-216.client.mchsi.com) Quit (Remote host closed the connection)
[0:40] * darkid (~Diablothe@9S0AAC1AW.tor-irc.dnsbl.oftc.net) Quit ()
[0:41] * theghost99 (~djidis__@exit2.blackcatz.org) has joined #ceph
[0:54] * rendar (~I@host190-216-dynamic.117-80-r.retail.telecomitalia.it) Quit ()
[0:54] * primechuck (~primechuc@173-17-128-216.client.mchsi.com) has joined #ceph
[1:10] * theghost99 (~djidis__@5NZAAFU3E.tor-irc.dnsbl.oftc.net) Quit ()
[1:11] * Salamander_ (~BlS@37.48.65.122) has joined #ceph
[1:14] * ilken (ilk@2602:63:c2a2:af00:c4c1:a93a:1b03:e8ec) Quit (Remote host closed the connection)
[1:15] * alejandrito (~alejandri@190.55.158.253) has joined #ceph
[1:16] * badone_ (~brad@CPE-121-215-241-179.static.qld.bigpond.net.au) has joined #ceph
[1:17] * i_m (~ivan.miro@88.206.99.242) Quit (Ping timeout: 480 seconds)
[1:22] * badone (~brad@CPE-121-215-241-179.static.qld.bigpond.net.au) Quit (Ping timeout: 480 seconds)
[1:40] * Salamander_ (~BlS@7R2AADCSB.tor-irc.dnsbl.oftc.net) Quit ()
[1:41] * andrew_m (~Mraedis@193.107.85.61) has joined #ceph
[2:01] * oms101 (~oms101@p20030057EA3A1F00EEF4BBFFFE0F7062.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[2:09] * oms101 (~oms101@p20030057EA0C7000EEF4BBFFFE0F7062.dip0.t-ipconnect.de) has joined #ceph
[2:11] * andrew_m (~Mraedis@9S0AAC1DZ.tor-irc.dnsbl.oftc.net) Quit ()
[2:11] * dicko (~Uniju@89.105.194.91) has joined #ceph
[2:24] * lucas1 (~Thunderbi@218.76.52.64) has joined #ceph
[2:30] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[2:30] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[2:33] * i_m (~ivan.miro@87.249.207.228) has joined #ceph
[2:33] * cholcombe (~chris@c-73-180-29-35.hsd1.or.comcast.net) Quit (Ping timeout: 480 seconds)
[2:41] * dicko (~Uniju@9S0AAC1EP.tor-irc.dnsbl.oftc.net) Quit ()
[2:41] * Dinnerbone (~darkid@162.247.73.206) has joined #ceph
[3:02] * primechuck (~primechuc@173-17-128-216.client.mchsi.com) Quit (Remote host closed the connection)
[3:07] * i_m (~ivan.miro@87.249.207.228) Quit (Ping timeout: 480 seconds)
[3:11] * Dinnerbone (~darkid@9S0AAC1FO.tor-irc.dnsbl.oftc.net) Quit ()
[3:11] * Aramande_ (~Jyron@216.218.134.12) has joined #ceph
[3:11] * zhaochao (~zhaochao@125.39.8.230) has joined #ceph
[3:17] * doppelgrau (~doppelgra@pd956d116.dip0.t-ipconnect.de) Quit (Quit: doppelgrau)
[3:36] * shang (~ShangWu@123-192-243-196.dynamic.kbronet.com.tw) has joined #ceph
[3:41] * Aramande_ (~Jyron@5NZAAFU83.tor-irc.dnsbl.oftc.net) Quit ()
[3:41] * AluAlu (~djidis__@89.105.194.87) has joined #ceph
[3:44] <Nats_> if i want to pre-emptively replace an OSD before it dies; if I dd'd the entire disk (partition talbe and all) would ceph accept the new disk as being equivalent to the one i removed?
[3:44] <gleam> yes, it should. i think you could even just rsync the data.
[3:45] <gleam> once you duplicate the partition table
[3:47] <Nats_> thats true, good idea
[3:47] <SamYaple> Nats_: if using btrfs you can even do a send/recv
[3:47] <gleam> good point
[3:48] <Nats_> xfs unfortunately
[3:48] <SamYaple> it would be unfortunate if you were using btrfs ;)
[3:48] <gleam> don't say unfortunately
[3:48] <gleam> right
[3:48] <gleam> you don't want to be using btrfs :(
[3:48] <SamYaple> but hey if you got it use it
[3:49] <Nats_> gleam, yeah agreed. just in this instance, unfortunately :)
[4:02] * nsoffer (~nsoffer@109.65.255.15) Quit (Ping timeout: 480 seconds)
[4:06] * shang (~ShangWu@123-192-243-196.dynamic.kbronet.com.tw) Quit (Ping timeout: 480 seconds)
[4:07] * flisky (~Thunderbi@106.39.60.34) has joined #ceph
[4:09] * haomaiwa_ (~haomaiwan@li1072-91.members.linode.com) has joined #ceph
[4:11] * AluAlu (~djidis__@7R2AADCW5.tor-irc.dnsbl.oftc.net) Quit ()
[4:11] * Esge (~arsenaali@7R2AADCX2.tor-irc.dnsbl.oftc.net) has joined #ceph
[4:13] * zz_kefu is now known as kefu
[4:15] * haomaiwang (~haomaiwan@li1072-91.members.linode.com) Quit (Ping timeout: 480 seconds)
[4:23] * OutOfNoWhere (~rpb@199.68.195.101) Quit (Ping timeout: 480 seconds)
[4:25] * haomaiwang (~haomaiwan@li1072-91.members.linode.com) has joined #ceph
[4:32] * haomaiwa_ (~haomaiwan@li1072-91.members.linode.com) Quit (Ping timeout: 480 seconds)
[4:33] * flisky (~Thunderbi@106.39.60.34) Quit (Quit: flisky)
[4:34] * yghannam (~yghannam@0001f8aa.user.oftc.net) Quit (Quit: Leaving)
[4:34] <edrian> morning
[4:35] <SamYaple> mrning edrian
[4:35] * shang (~ShangWu@175.41.48.77) has joined #ceph
[4:36] <edrian> the more I tried to fix my broken cluster, the more I fail doing so. I tired to change the crush map, the add more PGs. I reverted most of the last changes I've made but so far I'm in that state https://pastebin.mozilla.org/8841423
[4:41] * Esge (~arsenaali@7R2AADCX2.tor-irc.dnsbl.oftc.net) Quit ()
[4:41] * rf`1 (~Kizzi@104.255.64.26) has joined #ceph
[5:07] * Vacuum_ (~Vacuum@88.130.204.15) has joined #ceph
[5:11] * rf`1 (~Kizzi@7R2AADCY2.tor-irc.dnsbl.oftc.net) Quit ()
[5:11] * zviratko1 (~nartholli@104.255.64.26) has joined #ceph
[5:12] * alejandrito (~alejandri@190.55.158.253) Quit (Remote host closed the connection)
[5:14] * Vacuum__ (~Vacuum@88.130.211.54) Quit (Ping timeout: 480 seconds)
[5:23] * vbellur (~vijay@122.172.66.82) Quit (Ping timeout: 480 seconds)
[5:29] * as0bu (~as0bu@c-98-230-203-84.hsd1.nm.comcast.net) has joined #ceph
[5:29] * beardo_ (~sma310@207-172-244-241.c3-0.atw-ubr5.atw.pa.cable.rcn.com) Quit (Ping timeout: 480 seconds)
[5:34] * beardo_ (~sma310@207-172-244-241.c3-0.atw-ubr5.atw.pa.cable.rcn.com) has joined #ceph
[5:41] * zviratko1 (~nartholli@7R2AADCZ8.tor-irc.dnsbl.oftc.net) Quit ()
[5:41] * Solvius (~Vale@5.101.105.32) has joined #ceph
[5:49] * kanagaraj (~kanagaraj@117.216.107.84) has joined #ceph
[6:09] * rdas (~rdas@121.244.87.116) has joined #ceph
[6:11] * Solvius (~Vale@5NZAAFVDN.tor-irc.dnsbl.oftc.net) Quit ()
[6:11] * allenmelon (~Guest1390@cs-tor.bu.edu) has joined #ceph
[6:33] * ira (~ira@121.244.87.124) has joined #ceph
[6:36] * as0bu (~as0bu@c-98-230-203-84.hsd1.nm.comcast.net) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[6:41] * allenmelon (~Guest1390@7R2AADC14.tor-irc.dnsbl.oftc.net) Quit ()
[6:41] * ain (~Joppe4899@exit1.torproxy.org) has joined #ceph
[6:45] * derjohn_mob (~aj@tmo-112-176.customers.d1-online.com) Quit (Ping timeout: 480 seconds)
[6:50] * rotbeard (~redbeard@185.32.80.238) has joined #ceph
[7:00] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[7:00] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[7:02] * tserong (~tserong@203-214-92-220.dyn.iinet.net.au) Quit (Ping timeout: 480 seconds)
[7:03] * primechuck (~primechuc@173-17-128-216.client.mchsi.com) has joined #ceph
[7:04] * kefu is now known as zz_kefu
[7:11] * ain (~Joppe4899@9S0AAC1NN.tor-irc.dnsbl.oftc.net) Quit ()
[7:11] * LRWerewolf (~Freddy@195.169.125.226) has joined #ceph
[7:11] * primechuck (~primechuc@173-17-128-216.client.mchsi.com) Quit (Ping timeout: 480 seconds)
[7:18] * amote (~amote@121.244.87.116) has joined #ceph
[7:37] * kmARC (~kmARC@84-73-73-158.dclient.hispeed.ch) has joined #ceph
[7:38] * kanagaraj_ (~kanagaraj@117.197.189.76) has joined #ceph
[7:39] * kanagaraj (~kanagaraj@117.216.107.84) Quit (Ping timeout: 480 seconds)
[7:41] * LRWerewolf (~Freddy@5NZAAFVGX.tor-irc.dnsbl.oftc.net) Quit ()
[7:41] * Jones (~dusti@79.98.107.90) has joined #ceph
[7:45] * imcsk8 (~ichavero@189.231.89.185) Quit (Read error: Connection reset by peer)
[7:45] * imcsk8 (~ichavero@189.231.89.185) has joined #ceph
[7:47] * derjohn_mob (~aj@88.128.80.39) has joined #ceph
[7:50] * overclk (~overclk@121.244.87.117) has joined #ceph
[7:51] * pvh_sa_ (~pvh@105-237-39-21.access.mtnbusiness.co.za) Quit (Ping timeout: 480 seconds)
[7:55] * branto (~branto@178-253-132-71.3pp.slovanet.sk) has joined #ceph
[8:03] * pvh_sa_ (~pvh@105-237-39-21.access.mtnbusiness.co.za) has joined #ceph
[8:09] * nardial (~ls@dslb-178-006-188-098.178.006.pools.vodafone-ip.de) has joined #ceph
[8:11] * Jones (~dusti@7R2AADC4V.tor-irc.dnsbl.oftc.net) Quit ()
[8:11] * Pieman (~Defaultti@188.120.253.39) has joined #ceph
[8:22] * pvh_sa_ (~pvh@105-237-39-21.access.mtnbusiness.co.za) Quit (Ping timeout: 480 seconds)
[8:29] * dopesong (~dopesong@88-119-94-55.static.zebra.lt) has joined #ceph
[8:29] * T1w (~jens@node3.survey-it.dk) has joined #ceph
[8:30] * mookins (~mookins@induct3.lnk.telstra.net) has joined #ceph
[8:34] * zz_kefu is now known as kefu
[8:35] * b0e (~aledermue@213.95.25.82) has joined #ceph
[8:38] * kanagaraj_ (~kanagaraj@117.197.189.76) Quit (Quit: Leaving)
[8:38] * kanagaraj (~kanagaraj@117.197.189.76) has joined #ceph
[8:40] * derjohn_mob (~aj@88.128.80.39) Quit (Ping timeout: 480 seconds)
[8:41] * Pieman (~Defaultti@7R2AADC5J.tor-irc.dnsbl.oftc.net) Quit ()
[8:41] * _s1gma (~KapiteinK@173.255.226.142) has joined #ceph
[8:49] * kefu is now known as kefu|afk
[8:50] * kefu|afk is now known as zz_kefu|afk
[8:51] * shohn (~shohn@p57A14CE9.dip0.t-ipconnect.de) has joined #ceph
[8:58] * analbeard (~shw@support.memset.com) has joined #ceph
[8:58] * kawa2014 (~kawa@89.184.114.246) has joined #ceph
[8:58] * kmARC (~kmARC@84-73-73-158.dclient.hispeed.ch) Quit (Ping timeout: 480 seconds)
[9:00] * zz_kefu|afk is now known as kefu
[9:03] * rendar (~I@host224-193-dynamic.18-79-r.retail.telecomitalia.it) has joined #ceph
[9:03] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[9:05] * sleinen (~Adium@2a02:1205:34d7:1ac0:7ed1:c3ff:fedc:3223) has joined #ceph
[9:11] * _s1gma (~KapiteinK@5NZAAFVKH.tor-irc.dnsbl.oftc.net) Quit ()
[9:11] * tokie (~capitalth@tor-exit2-readme.puckey.org) has joined #ceph
[9:13] * ajazdzewski (~ajazdzews@lpz-66.sprd.net) has joined #ceph
[9:14] * Be-El (~quassel@fb08-bcf-pc01.computational.bio.uni-giessen.de) has joined #ceph
[9:14] <Be-El> hi
[9:17] * dgurtner (~dgurtner@178.197.231.188) has joined #ceph
[9:19] <ajazdzewski> hi
[9:20] <ajazdzewski> i curently facing some issues with ceph - we are try to bring up our new cluster with new hardware (del R730xd) and wenn made some test over the weekend
[9:21] <ajazdzewski> now i have somme messages like "1133 requests are blocked > 32 sec"
[9:22] <ajazdzewski> ceph health detail list a lot of osd how are 'slow'
[9:24] <ajazdzewski> the command grep 'slow request' /var/log/ceph/ceph.log | awk '{print $3}' | sort | uniq -c | sort -t'.' -k2h list all my osd in the cluster
[9:27] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[9:27] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[9:28] <ajazdzewski> the messages i found are multible times 'megasas:span 0 rowDataSize 1' so i will deeper into the controller
[9:30] * wicope (~wicope@0001fd8a.user.oftc.net) has joined #ceph
[9:32] * sleinen (~Adium@2a02:1205:34d7:1ac0:7ed1:c3ff:fedc:3223) Quit (Quit: Leaving.)
[9:35] * fsimonce (~simon@host249-48-dynamic.53-79-r.retail.telecomitalia.it) has joined #ceph
[9:40] * pvh_sa_ (~pvh@41.164.8.114) has joined #ceph
[9:41] * tokie (~capitalth@5NZAAFVLU.tor-irc.dnsbl.oftc.net) Quit ()
[9:41] * Architect (~Esge@37.48.65.122) has joined #ceph
[9:41] * derjohn_mob (~aj@fw.gkh-setu.de) has joined #ceph
[9:43] * karnan (~karnan@121.244.87.117) has joined #ceph
[9:46] * mookins (~mookins@induct3.lnk.telstra.net) Quit ()
[9:47] * jordanP (~jordan@213.215.2.194) has joined #ceph
[9:49] * brutuscat (~brutuscat@234.Red-79-151-98.dynamicIP.rima-tde.net) has joined #ceph
[9:55] * kmARC (~kmARC@2001:620:20:16:b058:4427:4aed:68e7) has joined #ceph
[9:56] * brutuscat (~brutuscat@234.Red-79-151-98.dynamicIP.rima-tde.net) Quit (Remote host closed the connection)
[9:56] * brutuscat (~brutuscat@234.Red-79-151-98.dynamicIP.rima-tde.net) has joined #ceph
[9:57] * pvh_sa_ (~pvh@41.164.8.114) Quit (Remote host closed the connection)
[10:00] * linjan_ (~linjan@176.195.232.220) has joined #ceph
[10:01] * daviddcc (~dcasier@77.151.197.84) has joined #ceph
[10:04] * ajazdzewski (~ajazdzews@lpz-66.sprd.net) Quit (Ping timeout: 480 seconds)
[10:04] * shang (~ShangWu@175.41.48.77) Quit (Ping timeout: 480 seconds)
[10:05] * foxdalas (~maximpogo@vpn1-ki-msk.ip.ngenix.net) has joined #ceph
[10:06] <foxdalas> re all
[10:06] <foxdalas> I have a problem with mds
[10:06] <foxdalas> log_channel(cluster) log [ERR] : failed to commit dir 1000000759c object, errno -2
[10:06] <foxdalas> 2015-08-03 08:02:53.306092 7f3703973700 -1 mds.0.99 unhandled write error (2) No such file or directory, force readonly...
[10:07] * karnan (~karnan@121.244.87.117) Quit (Ping timeout: 480 seconds)
[10:08] <foxdalas> Where is a directory or file ?
[10:11] * Architect (~Esge@5NZAAFVM0.tor-irc.dnsbl.oftc.net) Quit ()
[10:12] * kefu is now known as kefu|afk
[10:13] * kefu|afk is now known as kefu
[10:17] * karnan (~karnan@106.216.132.146) has joined #ceph
[10:22] * dopesong_ (~dopesong@lb1.mailer.data.lt) has joined #ceph
[10:22] * brutuscat (~brutuscat@234.Red-79-151-98.dynamicIP.rima-tde.net) Quit (Read error: Connection reset by peer)
[10:26] * TMM (~hp@178-84-46-106.dynamic.upc.nl) Quit (Quit: Ex-Chat)
[10:27] * brutuscat (~brutuscat@234.Red-79-151-98.dynamicIP.rima-tde.net) has joined #ceph
[10:29] * dopesong (~dopesong@88-119-94-55.static.zebra.lt) Quit (Ping timeout: 480 seconds)
[10:30] * jclm (~jclm@ip24-253-98-109.lv.lv.cox.net) has joined #ceph
[10:36] * shang (~ShangWu@175.41.48.77) has joined #ceph
[10:36] * doppelgrau (~doppelgra@pd956d116.dip0.t-ipconnect.de) has joined #ceph
[10:39] * jordanP (~jordan@213.215.2.194) Quit (Quit: Leaving)
[10:39] * shylesh (~shylesh@121.244.87.124) has joined #ceph
[10:41] * cyphase (~qable@exit2.blackcatz.org) has joined #ceph
[10:41] * cyphase is now known as Guest364
[10:42] * tom (~tom@167.88.45.146) Quit (Ping timeout: 480 seconds)
[10:44] * karnan (~karnan@106.216.132.146) Quit (Ping timeout: 480 seconds)
[10:51] * ajazdzewski (~ajazdzews@lpz-66.sprd.net) has joined #ceph
[10:52] * ajazdzewski (~ajazdzews@lpz-66.sprd.net) Quit ()
[10:52] <hlkv6-59469> Hi all, I have radosgw running and s3cmd working nicely, also Python working, but cannot make PHP test work :-(
[10:52] <hlkv6-59469> does anyone have a PHP sample I can try out?
[10:53] * ajazdzewski (~ajazdzews@lpz-66.sprd.net) has joined #ceph
[10:57] * karnan (~karnan@121.244.87.117) has joined #ceph
[11:01] * jks (~jks@178.155.151.121) Quit (Ping timeout: 480 seconds)
[11:02] * kutija (~kutija@89.216.27.139) has joined #ceph
[11:02] * jcsp (~jspray@82-71-16-249.dsl.in-addr.zen.co.uk) Quit (Ping timeout: 480 seconds)
[11:05] <kutija> I have a question - one of my storage nodes went down for a few hours and ceph health says
[11:05] <kutija> health HEALTH_WARN 3555 pgs degraded; 5824 pgs stuck unclean; recovery 217370/715324 objects degraded (30.388%); clock skew detected on mon.node-8, mon.node-9
[11:05] <kutija> and four of my OSD's are now down
[11:07] <kutija> the question is - what should I do to enable them and to sync them?
[11:07] * cooldharma06 (~chatzilla@14.139.180.40) has joined #ceph
[11:09] * Hemanth (~Hemanth@121.244.87.117) has joined #ceph
[11:11] * Guest364 (~qable@5NZAAFVPG.tor-irc.dnsbl.oftc.net) Quit ()
[11:11] * dug (~Scaevolus@relay-d.tor-exit.network) has joined #ceph
[11:12] <ajazdzewski> kutija: first of all have a look to you ntp use the same ntp servers on all your nodes
[11:12] <kutija> did that
[11:12] <kutija> that is dfixed
[11:13] <ajazdzewski> than have a look to "ceph health detail"
[11:14] <kutija> it gives a lot of
[11:14] <kutija> pg 3.14 is stuck unclean for 52671.263019, current state active+degraded, last acting [3]
[11:14] <kutija> and ofcourse
[11:14] <kutija> HEALTH_WARN 3555 pgs degraded; 5824 pgs stuck unclean; recovery 217371/715326 objects degraded (30.388%);
[11:15] <ajazdzewski> how many nodes and replicas do you have?
[11:15] <kutija> 2 nodes, 2 rplicas
[11:15] <ajazdzewski> ok that is a bit tricky
[11:16] <ajazdzewski> i sugest you to set min replica to 1
[11:17] <ajazdzewski> give me a moment i will look for the correct setting
[11:18] <ajazdzewski> osd pool default min size
[11:18] <kutija> and then?
[11:18] <ajazdzewski> http://ceph.com/docs/master/rados/troubleshooting/troubleshooting-pg/
[11:19] <ajazdzewski> your problem is that the defaul is 2 ans on your cluster only 1 copy is lefth
[11:19] <ajazdzewski> so schage this value to 1 and you cluster will start to recover
[11:21] * jordanP (~jordan@213.215.2.194) has joined #ceph
[11:21] * dgbaley27 (~matt@c-67-176-93-83.hsd1.co.comcast.net) Quit (Remote host closed the connection)
[11:22] * jcsp (~jspray@summerhall-meraki1.fluency.net.uk) has joined #ceph
[11:23] <kutija> well lets see
[11:26] <Be-El> if you only have two hosts and one of them is down, where do you think ceph should put the second copy to?
[11:27] * fdmanana (~fdmanana@bl13-153-23.dsl.telepac.pt) has joined #ceph
[11:31] <kutija> that setting is already at 1
[11:31] <kutija> osd_pool_default_min_size = 1
[11:32] <ajazdzewski> Two nodes is a bad idea for a cluster. Personality: you should start with 5 nodes as minimum
[11:32] <kutija> yes I know
[11:32] <kutija> but this is what I have and I need to solve this issue somehow
[11:33] <ajazdzewski> both nodes are up and fine?
[11:33] <kutija> yes, working fine
[11:33] <kutija> OSD's on the node that failed are down
[11:33] <ajazdzewski> ceph osd tree list all your osd
[11:33] <ajazdzewski> and all are up and in
[11:34] <kutija> http://pastebin.com/BgRczN1s
[11:34] <ajazdzewski> ok
[11:34] <ajazdzewski> so you have a reblica of 2
[11:34] <kutija> yes
[11:34] <ajazdzewski> the defaul crushmap ist that ceph wil place a copy of your data on a other host
[11:35] <ajazdzewski> all your osd on node 12 are down
[11:35] <ajazdzewski> so ceph wil no recover
[11:35] <ajazdzewski> because it find no other host wehre the data can be placed
[11:36] <ajazdzewski> you can edit the crushmap and allow to have replicas on OSD-level and not on Host-Level but that is bad
[11:37] <ajazdzewski> you can also set you replication level to 1 -> end up with the same risk
[11:37] <ajazdzewski> how fast you can bring up the node-12?
[11:38] <kutija> you mean get it online?
[11:38] <ajazdzewski> jep
[11:38] <kutija> it is online
[11:38] <kutija> this status
[11:38] <kutija> I've got it from that node
[11:38] <kutija> it's online and working
[11:38] <kutija> actually both of my servers are online
[11:38] <ajazdzewski> but the osds on node-12 are down
[11:39] <kutija> yea
[11:39] * TMM (~hp@sams-office-nat.tomtomgroup.com) has joined #ceph
[11:39] <kutija> but the node is operational
[11:40] <ajazdzewski> but will not serve data from the osd hosted on node-12
[11:40] <ajazdzewski> try a service ceph-all restart on your node-12
[11:40] <kutija> ceph-all: command not found
[11:41] * vikhyat (~vumrao@121.244.87.116) has joined #ceph
[11:41] * dug (~Scaevolus@7R2AADDBP.tor-irc.dnsbl.oftc.net) Quit ()
[11:41] * Rehevkor (~dicko@85.143.95.50) has joined #ceph
[11:44] <kutija> hm
[11:44] <kutija> no ceph processes are running
[11:47] <ajazdzewski> ok the command is on a ubuntu box
[11:47] <kutija> yeah I forgot about service part
[11:47] <kutija> ceph-all start/running
[11:47] <kutija> but
[11:48] <kutija> root@node-12:/var/log/ceph# ps uax | grep ceph
[11:48] <kutija> root 19142 0.0 0.0 11740 928 pts/1 S+ 09:47 0:00 grep --color=auto ceph
[11:48] <kutija> root@node-12:/var/log/ceph#
[11:49] <ajazdzewski> you can also try to start one osd manual in the forgound and look to the output
[11:53] <kutija> I've tried but I can't find what's wrong
[11:53] <kutija> and why it is not starting
[12:06] * brutuscat (~brutuscat@234.Red-79-151-98.dynamicIP.rima-tde.net) Quit (Read error: Connection reset by peer)
[12:08] <kutija> any ideas?
[12:11] * Rehevkor (~dicko@9S0AAC1YW.tor-irc.dnsbl.oftc.net) Quit ()
[12:11] * Xa (~AotC@185.61.149.51) has joined #ceph
[12:12] * brutuscat (~brutuscat@234.Red-79-151-98.dynamicIP.rima-tde.net) has joined #ceph
[12:15] * alexbligh1 (~alexbligh@89-16-176-215.no-reverse-dns-set.bytemark.co.uk) Quit (Ping timeout: 480 seconds)
[12:17] * alexbligh1 (~alexbligh@89-16-176-215.no-reverse-dns-set.bytemark.co.uk) has joined #ceph
[12:19] * jks (~jks@178.155.151.121) has joined #ceph
[12:23] * nardial (~ls@dslb-178-006-188-098.178.006.pools.vodafone-ip.de) Quit (Quit: Leaving)
[12:23] * jcsp (~jspray@summerhall-meraki1.fluency.net.uk) Quit (Quit: Ex-Chat)
[12:23] * T1w (~jens@node3.survey-it.dk) Quit (Ping timeout: 480 seconds)
[12:30] * wicope (~wicope@0001fd8a.user.oftc.net) Quit (Remote host closed the connection)
[12:31] * zhaochao (~zhaochao@125.39.8.230) Quit (Quit: ChatZilla 0.9.91.1 [Iceweasel 38.1.0/20150711212448])
[12:38] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[12:38] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[12:41] * Xa (~AotC@7R2AADDDI.tor-irc.dnsbl.oftc.net) Quit ()
[12:41] * cyphase (~Kyso@jaures.gtor.org) has joined #ceph
[12:41] * cyphase is now known as Guest378
[12:42] * ngoswami (~ngoswami@121.244.87.116) has joined #ceph
[12:43] * kanagaraj (~kanagaraj@117.197.189.76) Quit (Ping timeout: 480 seconds)
[12:47] * shang (~ShangWu@175.41.48.77) Quit (Quit: Ex-Chat)
[12:48] * kutija (~kutija@89.216.27.139) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[12:50] * kutija (~kutija@89.216.27.139) has joined #ceph
[12:54] * brutuscat (~brutuscat@234.Red-79-151-98.dynamicIP.rima-tde.net) Quit (Read error: Connection reset by peer)
[12:55] * brutuscat (~brutuscat@234.Red-79-151-98.dynamicIP.rima-tde.net) has joined #ceph
[12:57] <vikhyat> ping ceph any idea on radosgw pool name convention like is it necessary to have .rgw as prefix ?
[13:01] <kefu> as http://docs.ceph.com/docs/master/radosgw/config/#create-pools put.
[13:02] <kefu> you can use any name as you like.
[13:02] <vikhyat> kefu: right I think we can
[13:02] <vikhyat> no need to go with extension .rgw
[13:04] * brutuscat (~brutuscat@234.Red-79-151-98.dynamicIP.rima-tde.net) Quit (Remote host closed the connection)
[13:05] * overclk (~overclk@121.244.87.117) Quit (Quit: Leaving)
[13:11] * Guest378 (~Kyso@9S0AAC10R.tor-irc.dnsbl.oftc.net) Quit ()
[13:14] * thomnico (~thomnico@2a01:e35:8b41:120:5c49:d89a:ce49:4878) has joined #ceph
[13:16] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit (Ping timeout: 480 seconds)
[13:23] * beardo_ (~sma310@207-172-244-241.c3-0.atw-ubr5.atw.pa.cable.rcn.com) Quit (Ping timeout: 480 seconds)
[13:23] * dgurtner (~dgurtner@178.197.231.188) Quit (Ping timeout: 480 seconds)
[13:27] * dgurtner (~dgurtner@178.197.231.188) has joined #ceph
[13:35] * shylesh (~shylesh@121.244.87.124) Quit (Remote host closed the connection)
[13:44] * karnan (~karnan@121.244.87.117) Quit (Remote host closed the connection)
[13:45] * badone_ (~brad@CPE-121-215-241-179.static.qld.bigpond.net.au) Quit (Ping timeout: 480 seconds)
[13:50] * ganders (~root@190.2.42.21) has joined #ceph
[13:51] * dgurtner (~dgurtner@178.197.231.188) Quit (Ping timeout: 480 seconds)
[13:55] * nsoffer (~nsoffer@nat-pool-tlv-t.redhat.com) has joined #ceph
[13:55] * dgurtner (~dgurtner@178.197.231.188) has joined #ceph
[13:58] * kanagaraj (~kanagaraj@117.213.132.242) has joined #ceph
[14:01] * overclk (~overclk@117.202.111.189) has joined #ceph
[14:03] * brutuscat (~brutuscat@234.Red-79-151-98.dynamicIP.rima-tde.net) has joined #ceph
[14:06] * shohn1 (~shohn@p57A14CE9.dip0.t-ipconnect.de) has joined #ceph
[14:06] * shohn (~shohn@p57A14CE9.dip0.t-ipconnect.de) Quit (Read error: Connection reset by peer)
[14:08] * mookins (~mookins@27-32-204-26.static.tpgi.com.au) has joined #ceph
[14:09] * rdas (~rdas@121.244.87.116) Quit (Quit: Leaving)
[14:10] * mookins (~mookins@27-32-204-26.static.tpgi.com.au) Quit ()
[14:11] * darks (~Dragonsha@HabitedSerfdom.tor-exit.sec.gd) has joined #ceph
[14:12] * thomnico (~thomnico@2a01:e35:8b41:120:5c49:d89a:ce49:4878) Quit (Ping timeout: 480 seconds)
[14:13] * lucas1 (~Thunderbi@218.76.52.64) Quit (Remote host closed the connection)
[14:15] * kutija (~kutija@89.216.27.139) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[14:17] * jcsp (~jspray@summerhall-meraki1.fluency.net.uk) has joined #ceph
[14:26] * burley (~khemicals@cpe-98-28-239-78.cinci.res.rr.com) has joined #ceph
[14:30] * kanagaraj (~kanagaraj@117.213.132.242) Quit (Read error: Connection reset by peer)
[14:30] * kanagaraj (~kanagaraj@117.213.132.242) has joined #ceph
[14:34] * i_m (~ivan.miro@mail.iicmos.ru) has joined #ceph
[14:41] * darks (~Dragonsha@5NZAAFVXE.tor-irc.dnsbl.oftc.net) Quit ()
[14:41] * JohnO (~w2k@perry.fellwock.tor-exit.calyxinstitute.org) has joined #ceph
[14:42] * kefu is now known as zz_kefu
[14:49] * brutusca_ (~brutuscat@234.Red-79-151-98.dynamicIP.rima-tde.net) has joined #ceph
[14:49] * brutuscat (~brutuscat@234.Red-79-151-98.dynamicIP.rima-tde.net) Quit (Remote host closed the connection)
[14:51] * i_m (~ivan.miro@mail.iicmos.ru) Quit (Ping timeout: 480 seconds)
[14:55] * zz_kefu is now known as kefu
[14:55] <theanalyst> vikhyat: .rgw tends to be the default name that rgw uses when it creates the pools
[14:56] * primechuck (~primechuc@host-95-2-129.infobunker.com) has joined #ceph
[14:56] <vikhyat> theanalyst: yes right but if we want to create it manually I think we can create with any name
[14:57] <vikhyat> theanalyst: this I wanted to verify
[14:58] * branto (~branto@178-253-132-71.3pp.slovanet.sk) Quit (Quit: Leaving.)
[15:02] * overclk (~overclk@117.202.111.189) Quit (Remote host closed the connection)
[15:03] * ira (~ira@121.244.87.124) Quit (Ping timeout: 480 seconds)
[15:05] * i_m (~ivan.miro@deibp9eh1--blueice2n1.emea.ibm.com) has joined #ceph
[15:07] <dimir> burley: you were right, the size was the problem. Decreased the journal size and the script finished. :-)
[15:07] <dimir> thanks!
[15:08] * nsoffer (~nsoffer@nat-pool-tlv-t.redhat.com) Quit (Ping timeout: 480 seconds)
[15:09] * nsoffer (~nsoffer@nat-pool-tlv-t.redhat.com) has joined #ceph
[15:09] * overclk (~overclk@117.202.111.189) has joined #ceph
[15:11] * JohnO (~w2k@5NZAAFVYZ.tor-irc.dnsbl.oftc.net) Quit ()
[15:11] * TGF (~Wizeon@tor-exit0-readme.dfri.se) has joined #ceph
[15:11] * primechuck (~primechuc@host-95-2-129.infobunker.com) Quit (Quit: Leaving...)
[15:14] <SamYaple> 14
[15:14] * kanagaraj (~kanagaraj@117.213.132.242) Quit (Quit: Leaving)
[15:16] * tupper (~tcole@173.38.117.89) has joined #ceph
[15:17] * primechuck (~primechuc@host-95-2-129.infobunker.com) has joined #ceph
[15:22] * kutija (~kutija@89.216.27.139) has joined #ceph
[15:23] * haomaiwa_ (~haomaiwan@li817-120.members.linode.com) has joined #ceph
[15:24] <theanalyst> vikhyat: sure.. it does allow that
[15:25] * brad_mssw (~brad@66.129.88.50) has joined #ceph
[15:26] <burley> dimir: You're welcome
[15:28] <vikhyat> theanalyst: thanks
[15:28] * haomaiwang (~haomaiwan@li1072-91.members.linode.com) Quit (Remote host closed the connection)
[15:30] * kefu is now known as kefu|afk
[15:30] <kutija> I've found what is wrong with my OSD's
[15:30] <kutija> 2015-08-03 13:23:02.111918 7fee87334800 -1 filestore(/var/lib/ceph/osd/ceph-0) mount failed to open journal /var/lib/ceph/osd/ceph-0/journal: (2) No such file or directory
[15:30] <kutija> somehow, journal is a symlink to this
[15:30] <kutija> lrwxrwxrwx 1 root root 67 Jun 2 19:31 journal -> /dev/disk/by-id/ata-Samsung_SSD_850_PRO_512GB_S2BENWAG204585N-part4
[15:31] <kutija> but for whatever reason that does not exist anymore, only
[15:31] <kutija> lrwxrwxrwx 1 root root 9 Aug 3 08:54 ata-Samsung_SSD_850_PRO_512GB_S2BENWAG204585N -> ../../sda
[15:31] <kutija> so basically my journal is in trouble on a broken node
[15:31] <kutija> that's the reason my OSD's are not working
[15:31] <kutija> and the question is how to repair it
[15:32] <m0zes> if the journal partitions are gone, there is a good chance you'll just want to re-create the osds on that node. you'll also want to figure out how that ssd lost its partitions.
[15:35] * kiwnix (~egarcia@00011f91.user.oftc.net) Quit (Ping timeout: 480 seconds)
[15:35] * brutusca_ (~brutuscat@234.Red-79-151-98.dynamicIP.rima-tde.net) Quit (Read error: Connection reset by peer)
[15:35] <kutija> that is actually pretty good question
[15:38] * nsoffer (~nsoffer@nat-pool-tlv-t.redhat.com) Quit (Quit: Segmentation fault (core dumped))
[15:38] <m0zes> there are tools to recreate the journals, but if they weren't flushed properly there are probably missing writes on the osds.
[15:41] * TGF (~Wizeon@9S0AAC162.tor-irc.dnsbl.oftc.net) Quit ()
[15:41] * brutuscat (~brutuscat@234.Red-79-151-98.dynamicIP.rima-tde.net) has joined #ceph
[15:41] * dneary (~dneary@50-205-29-3-static.hfc.comcastbusiness.net) has joined #ceph
[15:41] * Coe|work (~legion@195.228.45.176) has joined #ceph
[15:41] <kutija> so basically I should re-create journal from scratch
[15:41] <kutija> and OSD's
[15:41] <kutija> and then hope that everything will be fine
[15:44] <m0zes> hopefully. and maybe get a replacement ssd in soon ;)
[15:44] * cooldharma06 (~chatzilla@14.139.180.40) Quit (Quit: ChatZilla 0.9.91.1 [Iceweasel 21.0/20130515140136])
[15:45] <kutija> well it's 1.5 months old :)
[15:45] <kutija> and judging by SMART status it's fine
[15:48] <m0zes> sure, but it has now lost partitions (somehow). if you didn't do it on purpose, something killed them. I'm more inclined to believe that the drive is failing
[15:56] * rotbeard (~redbeard@185.32.80.238) Quit (Quit: Leaving)
[15:57] * overclk (~overclk@117.202.111.189) Quit (Remote host closed the connection)
[16:01] * haomaiwa_ (~haomaiwan@li817-120.members.linode.com) Quit (Remote host closed the connection)
[16:01] * overclk (~overclk@117.202.111.189) has joined #ceph
[16:01] * shaunm (~shaunm@50-5-225-172.dynamic.fuse.net) has joined #ceph
[16:01] * thomnico (~thomnico@2a01:e35:8b41:120:3900:513d:e487:aeaa) has joined #ceph
[16:02] * haomaiwang (~haomaiwan@li817-120.members.linode.com) has joined #ceph
[16:03] * brutuscat (~brutuscat@234.Red-79-151-98.dynamicIP.rima-tde.net) Quit (Read error: Connection reset by peer)
[16:08] * danieagle (~Daniel@187.75.19.50) has joined #ceph
[16:09] * dneary (~dneary@50-205-29-3-static.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[16:11] * Coe|work (~legion@5NZAAFV1S.tor-irc.dnsbl.oftc.net) Quit ()
[16:11] * rikai1 (~roaet@188.120.253.39) has joined #ceph
[16:12] * erice (~erice@c-76-120-53-165.hsd1.co.comcast.net) Quit (Remote host closed the connection)
[16:12] * thomnico (~thomnico@2a01:e35:8b41:120:3900:513d:e487:aeaa) Quit (Quit: Ex-Chat)
[16:13] * erice (~erice@c-76-120-53-165.hsd1.co.comcast.net) has joined #ceph
[16:13] * erice (~erice@c-76-120-53-165.hsd1.co.comcast.net) Quit (Remote host closed the connection)
[16:13] * brutuscat (~brutuscat@234.Red-79-151-98.dynamicIP.rima-tde.net) has joined #ceph
[16:20] * vikhyat (~vumrao@121.244.87.116) Quit (Quit: Leaving)
[16:22] * thomnico (~thomnico@2a01:e35:8b41:120:243c:99ba:52f8:49d5) has joined #ceph
[16:30] * foxdalas (~maximpogo@vpn1-ki-msk.ip.ngenix.net) Quit (Quit: This computer has gone to sleep)
[16:30] * vbellur (~vijay@122.172.220.185) has joined #ceph
[16:31] * TheSov (~TheSov@cip-248.trustwave.com) has joined #ceph
[16:31] <TheSov> how would i add a new osd without using ceph-deploy?
[16:33] <kefu|afk> TheSov, http://docs.ceph.com/docs/master/rados/operations/add-or-rm-osds/
[16:33] * kefu|afk is now known as kefu
[16:33] <TheSov> thanks
[16:33] <kefu> yw
[16:36] * overclk (~overclk@117.202.111.189) Quit (Remote host closed the connection)
[16:39] * yghannam (~yghannam@0001f8aa.user.oftc.net) has joined #ceph
[16:41] * rikai1 (~roaet@9S0AAC2AA.tor-irc.dnsbl.oftc.net) Quit ()
[16:43] * Hemanth (~Hemanth@121.244.87.117) Quit (Ping timeout: 480 seconds)
[16:44] * sleinen (~Adium@172.113.77.83.dynamic.wline.res.cust.swisscom.ch) has joined #ceph
[16:46] * jbautista- (~wushudoin@38.140.108.2) has joined #ceph
[16:46] * sleinen1 (~Adium@2001:620:0:82::101) has joined #ceph
[16:49] <TheSov> so if an OSD fails outright does the cluster automatically backfill to correct?
[16:49] <TheSov> or do i have to delete the osd?
[16:50] <burley> after a default delay it'll auto-backfill
[16:51] <TheSov> how long we talking here?
[16:51] <burley> mon osd down out interval
[16:51] <TheSov> an hour a day?
[16:51] <burley> not sure what the default it, I think its a few minutes
[16:51] <TheSov> oh nice
[16:51] <burley> we set to 600 (10m)
[16:51] <TheSov> ok but if its down, why does the monitor show stale buckets even days later?
[16:52] * nsoffer (~nsoffer@nat-pool-tlv-t.redhat.com) has joined #ceph
[16:52] <burley> not sure, it shouldn't if all is well unless the backfill is just taking forever
[16:53] * sleinen (~Adium@172.113.77.83.dynamic.wline.res.cust.swisscom.ch) Quit (Ping timeout: 480 seconds)
[16:53] <TheSov> so in all seriousness if i have a truly large cluster, i can basically ignore disk failures for a bit of time before replacing them? im assuming the cluster shrinks to accomadate the loss of disks
[16:56] <burley> yes, we typically don't replace a disk until we have had a few fail
[16:57] * wicope (~wicope@0001fd8a.user.oftc.net) has joined #ceph
[16:57] * shakamunyi (~shakamuny@c-67-180-191-38.hsd1.ca.comcast.net) has joined #ceph
[16:57] * kutija (~kutija@89.216.27.139) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[16:58] * ajazdzewski (~ajazdzews@lpz-66.sprd.net) Quit (Quit: Konversation terminated!)
[16:58] * joshd1 (~jdurgin@68-119-140-18.dhcp.ahvl.nc.charter.com) has joined #ceph
[16:59] * MACscr|lappy (~MACscrlap@96.90.110.84) has joined #ceph
[17:00] * jrocha (~jrocha@vagabond.cern.ch) has joined #ceph
[17:00] * zaitcev (~zaitcev@c-76-113-49-212.hsd1.nm.comcast.net) has joined #ceph
[17:01] * haomaiwang (~haomaiwan@li817-120.members.linode.com) Quit (Read error: Connection reset by peer)
[17:02] * haomaiwang (~haomaiwan@li817-120.members.linode.com) has joined #ceph
[17:02] * kutija (~kutija@89.216.27.139) has joined #ceph
[17:03] * analbeard (~shw@support.memset.com) Quit (Quit: Leaving.)
[17:06] * as0bu (~as0bu@c-98-230-203-84.hsd1.nm.comcast.net) has joined #ceph
[17:09] * shakamunyi (~shakamuny@c-67-180-191-38.hsd1.ca.comcast.net) Quit (Remote host closed the connection)
[17:11] * richardus1 (~Snowman@162.247.72.212) has joined #ceph
[17:14] * shakamunyi (~shakamuny@c-67-180-191-38.hsd1.ca.comcast.net) has joined #ceph
[17:16] * sleinen1 (~Adium@2001:620:0:82::101) Quit (Ping timeout: 480 seconds)
[17:16] * reed (~reed@75-101-54-131.dsl.static.fusionbroadband.com) has joined #ceph
[17:19] * b0e (~aledermue@213.95.25.82) Quit (Quit: Leaving.)
[17:20] * jbautista- (~wushudoin@38.140.108.2) Quit (Remote host closed the connection)
[17:21] * thomnico (~thomnico@2a01:e35:8b41:120:243c:99ba:52f8:49d5) Quit (Ping timeout: 480 seconds)
[17:22] * as0bu (~as0bu@c-98-230-203-84.hsd1.nm.comcast.net) Quit (Quit: Textual IRC Client: www.textualapp.com)
[17:23] * wushudoin (~wushudoin@38.140.108.2) has joined #ceph
[17:23] * shakamunyi (~shakamuny@c-67-180-191-38.hsd1.ca.comcast.net) Quit (Remote host closed the connection)
[17:23] <TheSov> yeah i just verified that the cluster availability shrinks
[17:23] <TheSov> thats amazing
[17:24] * as0bu (~as0bu@c-98-230-203-84.hsd1.nm.comcast.net) has joined #ceph
[17:25] <TheSov> do i have to do anything to re-in the down osds?
[17:27] * dopesong (~dopesong@88-119-94-55.static.zebra.lt) has joined #ceph
[17:27] * davidz (~davidz@2605:e000:1313:8003:bca6:a94d:e9bc:63d4) has joined #ceph
[17:27] * dopeson__ (~dopesong@88-119-94-55.static.zebra.lt) has joined #ceph
[17:28] * thomnico (~thomnico@2a01:e35:8b41:120:5194:ce76:9ebe:30) has joined #ceph
[17:29] <TheSov> osdmap e48: 4 osds: 3 up, 3 in
[17:29] <TheSov> . does anyone know what i have to do get to get 4 up and 4 in?
[17:30] <burley> TheSov: There are differing approaches to restoring full service, we just delete the downed OSDs and create a new one when we replace it
[17:30] <nyov> after upgrading ceph debian python-ceph package to 0.94.2 bpo, there seems to be no more python ceph_argparse package bundled?
[17:30] <burley> in some cases that isn't necessary, but by using this method we have one consistent method to manage a bunch of cases
[17:31] <nyov> after running "ceph auth get ..."
[17:31] <nyov> Traceback (most recent call last):
[17:31] <nyov> File "/usr/bin/ceph", line 74, in <module>
[17:31] <nyov> from ceph_argparse import \
[17:31] <nyov> ImportError: No module named ceph_argparse
[17:31] <TheSov> well i only had 1 osd on this machine and i just turned it off
[17:33] * dopesong_ (~dopesong@lb1.mailer.data.lt) Quit (Ping timeout: 480 seconds)
[17:35] * dopesong (~dopesong@88-119-94-55.static.zebra.lt) Quit (Ping timeout: 480 seconds)
[17:35] * yguang11 (~yguang11@66.228.162.44) has joined #ceph
[17:35] <TheSov> i can "ceph osd in" it correct?
[17:35] * dopeson__ (~dopesong@88-119-94-55.static.zebra.lt) Quit (Ping timeout: 480 seconds)
[17:39] * xarses_ (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[17:40] * shakamunyi (~shakamuny@c-67-180-191-38.hsd1.ca.comcast.net) has joined #ceph
[17:41] * richardus1 (~Snowman@7R2AADDPW.tor-irc.dnsbl.oftc.net) Quit ()
[17:41] * BlS (~Silentkil@95.128.43.164) has joined #ceph
[17:44] * cholcombe (~chris@c-73-180-29-35.hsd1.or.comcast.net) has joined #ceph
[17:48] <TheSov> ok so doing it manually is a nightmare, its way easier to use ceph-deploy osd prepare and activeate for a new disk
[17:48] * moore (~moore@64.202.160.88) has joined #ceph
[17:48] <TheSov> 106 active+clean
[17:48] <TheSov> 73 active+degraded
[17:48] <TheSov> 13 active+recovering+degraded
[17:48] <TheSov> recovery io 38183 kB/s, 10 objects/s
[17:49] * wicope (~wicope@0001fd8a.user.oftc.net) Quit (Read error: No route to host)
[17:51] <TheSov> i wish ceph had an auto replace type deally
[17:52] <TheSov> the whole delete a osd and re-create it thing is too complex for business's today, they dont want to pay a 6 figure salary for a storage admin to change disks, they want interns to do that
[17:55] * sleinen (~Adium@172.113.77.83.dynamic.wline.res.cust.swisscom.ch) has joined #ceph
[17:56] <burley> thesov: We have it all scripted up to a single command
[17:56] * sleinen1 (~Adium@2001:620:0:69::103) has joined #ceph
[17:58] <TheSov> oh?
[17:58] <rkeene> Ours is automated -- if an OSD dies it's ejected after some testing, and any time you add a disk it's added as an OSD
[17:58] <TheSov> mind sharing?
[17:59] <TheSov> help a brutha out guys! how about some shelltastic sharing?
[18:00] <rkeene> I'm not sure it's too useful to anyone but us, and also the license isn't permissive so you can't do too much with it, but you can see what we do here: https://aurae.coreadaptive.com/artifact/4a413fccb8bd953d
[18:00] * sleinen (~Adium@172.113.77.83.dynamic.wline.res.cust.swisscom.ch) Quit (Read error: Connection reset by peer)
[18:01] * haomaiwang (~haomaiwan@li817-120.members.linode.com) Quit (Remote host closed the connection)
[18:01] <rkeene> Probably give you enough ideas to write your own independent version that does the same sort of thing
[18:01] <TheSov> thanks rkeene i was thinking of possibly setting up a website using php and python
[18:02] * kutija is now known as kutija|away
[18:02] * haomaiwang (~haomaiwan@li817-120.members.linode.com) has joined #ceph
[18:02] * kutija|away (~kutija@89.216.27.139) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[18:02] <rkeene> Ours is just an automated daemon that checks every 10 minutes
[18:02] <TheSov> it checks for new disks and down disks?
[18:02] * Be-El (~quassel@fb08-bcf-pc01.computational.bio.uni-giessen.de) Quit (Remote host closed the connection)
[18:03] * bitserker (~toni@188.87.126.67) has joined #ceph
[18:03] <rkeene> That part only checks for new disks
[18:04] <TheSov> ok im really really scared of losing a journal ssd and having a bunch of osd's die, by default crush will not allow replica data on the same host correct?
[18:04] <rkeene> There's another one that checks to see if an OSD is "broken" and will kick it out after testing the filesystem, but it's not pushed yet
[18:04] <rkeene> Right
[18:04] <rkeene> So if you lose a host you'll have a replica (we do 1 master, 2 replicas)
[18:04] * sleinen1 (~Adium@2001:620:0:69::103) Quit (Read error: Connection reset by peer)
[18:05] <TheSov> so your size is 3
[18:05] <TheSov> ?
[18:05] <rkeene> Once there are more than 3 monitors our size is 3, until then it mirrors the disks using Linux metadevices (RAID1)
[18:05] <rkeene> It's complicated
[18:05] <rkeene> Err, 3 or more monitors
[18:06] * i_m (~ivan.miro@deibp9eh1--blueice2n1.emea.ibm.com) Quit (Ping timeout: 480 seconds)
[18:06] <rkeene> (We used to use BtrFS and BtrFS mirroring, but it was too slow)
[18:06] <rkeene> https://aurae.coreadaptive.com/info/8672087059d342c11f4cdbd862cc95fb80857288?ln=0 (btrfs'd version)
[18:06] * bitserker (~toni@188.87.126.67) Quit ()
[18:07] <TheSov> I was thinking a tiered zfs system but it would be expensive
[18:07] <rkeene> Hmm ?
[18:07] <TheSov> so bsd with zfs + iscsi to osd server, put like 5 or 6 of them
[18:07] <TheSov> per osd server
[18:07] <[arx]> need user/password :}
[18:07] <rkeene> [arx], D'oh
[18:07] <rkeene> https://aurae.coreadaptive.com/artifact/8672087059d342c1
[18:08] * [arx] tried rkeene/hunter2
[18:08] <TheSov> LOL
[18:08] <TheSov> ah yes the old hunter2 joke
[18:08] <rkeene> "*******" is only my master password -- I hash all my passwords, http://webpass.rkeene.org/
[18:09] * TMM (~hp@sams-office-nat.tomtomgroup.com) Quit (Ping timeout: 480 seconds)
[18:10] <rkeene> (Pure javascript password manager, which never stores or transmits any passwords)
[18:10] <TheSov> so yes you get a bunch of bsd/zfs to iscsi servers, map them 5 at a time to a OSD server, and put that in ceph
[18:11] * BlS (~Silentkil@9S0AAC2FC.tor-irc.dnsbl.oftc.net) Quit ()
[18:11] * Azru (~Freddy@marcuse-2.nos-oignons.net) has joined #ceph
[18:11] <TheSov> so you have zfs backend passing to ceph
[18:12] <[arx]> sounds complex
[18:12] <TheSov> its not
[18:12] <rkeene> Yeah, but what would be the advantage over just using ZFS on the OSD ?
[18:12] * MACscr|lappy (~MACscrlap@96.90.110.84) Quit (Ping timeout: 480 seconds)
[18:12] * jcsp (~jspray@summerhall-meraki1.fluency.net.uk) Quit (Ping timeout: 480 seconds)
[18:12] <TheSov> basically storage servers, install freenas, export disk as iscsi. linux servers attach iscsi, connect to cephj
[18:12] <TheSov> ZOL sucks
[18:13] <rkeene> And Ceph won't run on FreeBSD/Solaris ?
[18:13] <[arx]> nope
[18:14] * erice (~erice@c-76-120-53-165.hsd1.co.comcast.net) has joined #ceph
[18:14] * dopesong (~dopesong@78-60-74-130.static.zebra.lt) has joined #ceph
[18:14] <TheSov> zfs on bsd is amazing, like art and science merged into a beautiful sculpture, ZOL is like someone took that sculpture smashed it into a million peices and used bluetack to put it back together.
[18:15] <[arx]> s/complex/fragile/
[18:15] * lcurtis (~lcurtis@47.19.105.250) has joined #ceph
[18:15] * ganders (~root@190.2.42.21) Quit (Quit: WeeChat 0.4.2)
[18:15] <TheSov> [arx], its like you're reading my mind.
[18:16] * thomnico (~thomnico@2a01:e35:8b41:120:5194:ce76:9ebe:30) Quit (Quit: Ex-Chat)
[18:16] * thomnico (~thomnico@2a01:e35:8b41:120:5194:ce76:9ebe:30) has joined #ceph
[18:17] <TheSov> I would love if someone would compile ceph for bsd
[18:17] <TheSov> it would be like a dream come true
[18:18] <rkeene> Where does it fail now ?
[18:18] <TheSov> no idea ive never tried, but i know others haved, and failed
[18:18] <rkeene> I'm mostly too lazy to deal with it, but does it not even work with the Linux personality ?
[18:18] <[arx]> https://wiki.ceph.com/Planning/Blueprints/Emperor/Increasing_Ceph_portability
[18:19] <[arx]> was the last thing i heard about it
[18:20] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[18:20] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Remote host closed the connection)
[18:20] <rkeene> Doesn't look too terrible to port
[18:21] <TheSov> i tried to run ceph on the raspi but it looks like hammer is not available as maintainers gave up on ceph arm
[18:21] * t0rn (~ssullivan@2607:fad0:32:a02:56ee:75ff:fe48:3bd3) has joined #ceph
[18:21] <rkeene> AIO is probably out (FreeBSD equiv. ?), libuuid might not be compatible and require some work
[18:21] * t0rn (~ssullivan@2607:fad0:32:a02:56ee:75ff:fe48:3bd3) has left #ceph
[18:22] * erice (~erice@c-76-120-53-165.hsd1.co.comcast.net) Quit (Ping timeout: 480 seconds)
[18:22] * MACscr|lappy (~MACscrlap@96.90.110.84) has joined #ceph
[18:24] * kefu is now known as kefu|afk
[18:25] * kefu|afk is now known as kefu
[18:25] * brutuscat (~brutuscat@234.Red-79-151-98.dynamicIP.rima-tde.net) Quit (Read error: No route to host)
[18:25] * Nacer_ (~Nacer@203-206-190-109.dsl.ovh.fr) has joined #ceph
[18:26] * brutuscat (~brutuscat@234.Red-79-151-98.dynamicIP.rima-tde.net) has joined #ceph
[18:27] * dgurtner (~dgurtner@178.197.231.188) Quit (Ping timeout: 480 seconds)
[18:29] * Nacer (~Nacer@2001:41d0:fe82:7200:8109:e588:e133:77db) Quit (Ping timeout: 480 seconds)
[18:33] * joshd1 (~jdurgin@68-119-140-18.dhcp.ahvl.nc.charter.com) Quit (Quit: Leaving.)
[18:34] * wicope (~wicope@0001fd8a.user.oftc.net) has joined #ceph
[18:37] * ira (~ira@1.186.32.22) has joined #ceph
[18:37] * jordanP (~jordan@213.215.2.194) Quit (Quit: Leaving)
[18:39] * kawa2014 (~kawa@89.184.114.246) Quit (Quit: Leaving)
[18:40] * primechuck (~primechuc@host-95-2-129.infobunker.com) Quit (Remote host closed the connection)
[18:41] * Azru (~Freddy@7R2AADDSD.tor-irc.dnsbl.oftc.net) Quit ()
[18:41] * kefu is now known as zz_kefu
[18:41] * PuyoDead (~Sophie@3.tor.exit.babylon.network) has joined #ceph
[18:43] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[18:43] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[18:44] * ircolle (~ircolle@mobile-166-173-060-136.mycingular.net) has joined #ceph
[18:47] * zz_kefu is now known as kefu
[18:48] * primechuck (~primechuc@host-95-2-129.infobunker.com) has joined #ceph
[18:48] * kutija (~kutija@95.180.90.38) has joined #ceph
[18:48] * jwilkins (~jowilkin@c-67-180-123-48.hsd1.ca.comcast.net) has joined #ceph
[18:50] * nsoffer (~nsoffer@nat-pool-tlv-t.redhat.com) Quit (Ping timeout: 480 seconds)
[18:50] * thomnico (~thomnico@2a01:e35:8b41:120:5194:ce76:9ebe:30) Quit (Quit: Ex-Chat)
[18:53] * thomnico (~thomnico@2a01:e35:8b41:120:5194:ce76:9ebe:30) has joined #ceph
[18:54] <kefu> TheSov, at least there is an osx port under development https://github.com/ceph/ceph/tree/port/new
[18:54] <TheSov> kefu, thats great but i would never run an apple server
[18:55] <kefu> yeah, but osx is sort of a BSD descendant in userspace
[18:56] <TheSov> if you wanna get technical sure
[18:56] <TheSov> i just dont see it that way
[18:56] * dgurtner (~dgurtner@178.197.224.22) has joined #ceph
[18:56] <kefu> so, if `#if defined(__APPLE__)` works, i'd say there are good chances that `#if defined(__FreeBSD__)` is close.
[18:57] <kefu> probably i am too optimistic =)
[18:57] <TheSov> who knows, im not a programmer. but I do know in terms of servers and how many disk bays they have, apple is no where near any functionality
[18:58] <TheSov> business's would use
[18:58] <kefu> i understand.
[18:58] <TheSov> so yeah it is a step in the right direction, i agree
[18:58] <kefu> that's my point =D
[18:58] <TheSov> yeah now we just need an arm port
[18:59] <TheSov> I want to build and sell OSD modules
[18:59] <TheSov> but x86 is too expensive for 1 osd per unit system
[18:59] <kefu> ceph does support ARM, right?
[18:59] <TheSov> it does but they have no builder for arm
[19:00] <TheSov> they are looking for one
[19:00] <kefu> oh, yeah.
[19:00] <kefu> no even crossbuild builder.
[19:00] <TheSov> look at it like this, if we had an arm builder, you could buy a synology nas and possibly load ceph on it
[19:00] <rkeene> That sounds terrible.
[19:00] <kefu> haha.
[19:00] <TheSov> nowhy?
[19:01] <TheSov> 4 osds for like 300 bucks + disks
[19:01] * haomaiwang (~haomaiwan@li817-120.members.linode.com) Quit (Remote host closed the connection)
[19:01] <rkeene> The memory requirements of Ceph and most likely the I/O capabilities of the device (which I haven't looked up)
[19:01] <TheSov> 1 gig per TB?
[19:01] <TheSov> the point is a scaleable system on the very cheap
[19:02] * haomaiwang (~haomaiwan@li817-120.members.linode.com) has joined #ceph
[19:02] <TheSov> https://www.linkedin.com/grp/post/6577799-5984198086061215744 <--- once these guys finish these, i can finally ditch my san
[19:02] <rkeene> How much RAM does that device have ?
[19:02] <TheSov> its variable
[19:02] <TheSov> the 4 bay unit has 2gb
[19:02] <rkeene> So 2GB would really be enough for about 1 OSD
[19:02] <TheSov> but they have different units per config
[19:03] <rkeene> (My 1TB OSD is using 1.4GB right now: root 13153 4.7 2.1 5854944 1433692 ? Sl Jun09 3738:19 /bin/ceph-osd -i 0 --pid-file /var/run/ceph/osd.0.pid -c /etc/ceph/ceph.conf --cluster ceph -f )
[19:03] <TheSov> https://www.synology.com/en-us/products/DS2015xs <-- upto 8gb
[19:04] <TheSov> you are cacheing too many pg maps
[19:04] <TheSov> turn it down
[19:04] <rkeene> The performance is already terrible
[19:04] <TheSov> you think performance will go down caching less maps?
[19:05] <TheSov> cern identified it as one of the biggest wastes of memory
[19:05] <rkeene> CERN also decided Ceph was usable, which I've yet to see
[19:05] <TheSov> each osd will cache like 500 of these maps
[19:05] <TheSov> they are using it now
[19:05] <TheSov> they use ceph to drive their openstack environment
[19:06] <rkeene> I only have 64 placement groups, so there's not a lot to cache
[19:06] <TheSov> and recently we found out that sourceforge had been using ceph, incorrectly, all this time
[19:06] <rkeene> Incorrectly ?
[19:06] <rkeene> What were they doing wrong ?
[19:07] * Aemerson (~Icedove@aa2.linuxbox.com) has joined #ceph
[19:07] <TheSov> they didnt admit to anything yet, its my opinion that i interject. I believe there were some misconfigured osd servers that caused data corruption for them
[19:07] <TheSov> but their cluster failed with corrupt data
[19:08] <rkeene> It could be a bug in Ceph
[19:08] <rkeene> Without knowing anything it's definitely a step too far to say they were using Ceph incorrectly
[19:11] * PuyoDead (~Sophie@7R2AADDTS.tor-irc.dnsbl.oftc.net) Quit ()
[19:11] * Xa (~Shnaw@5NZAAFWCZ.tor-irc.dnsbl.oftc.net) has joined #ceph
[19:12] <TheSov> from what ive been reading people are using ceph in ways not really supported and they are fine, my assumption is they never disabled the disk write cache in linux
[19:13] <TheSov> they must have had some issue which led to the corruption, we also do not know if it was RBD or cephfs
[19:13] * dgurtner (~dgurtner@178.197.224.22) Quit (Read error: Connection reset by peer)
[19:13] * kmARC (~kmARC@2001:620:20:16:b058:4427:4aed:68e7) Quit (Ping timeout: 480 seconds)
[19:14] * shaunm (~shaunm@50-5-225-172.dynamic.fuse.net) Quit (Ping timeout: 480 seconds)
[19:19] * BManojlovic (~steki@cable-89-216-173-148.dynamic.sbb.rs) has joined #ceph
[19:21] <burley> TheSov: That is incorrect conjecture wrt SF.net
[19:21] <burley> we'll provide more details once everything is restored
[19:22] * nsoffer (~nsoffer@nat-pool-tlv-t1.redhat.com) has joined #ceph
[19:23] * rotbeard (~redbeard@2a02:908:df18:6480:76f0:6dff:fe3b:994d) has joined #ceph
[19:26] * vikhyat (~vumrao@49.248.203.182) has joined #ceph
[19:27] * overclk (~overclk@117.202.111.189) has joined #ceph
[19:28] * overclk (~overclk@117.202.111.189) Quit ()
[19:28] <burley> TheSov: For the already publicly available details, see: http://tracker.ceph.com/issues/11586
[19:31] * nsoffer (~nsoffer@nat-pool-tlv-t1.redhat.com) Quit (Ping timeout: 480 seconds)
[19:31] * nsoffer (~nsoffer@nat-pool-tlv-t.redhat.com) has joined #ceph
[19:32] <TheSov> ok so its increasing the log level that did this
[19:32] <TheSov> i would say that incorrect usage
[19:33] <TheSov> and they are a couple versions behind
[19:33] <TheSov> interesting quote "This sounds exactly like an issue we've seen before with I think a certain combination of XFS version and Ceph's use of some less common filesystem features. It's definitely not Ceph doing this but the local FS. Have you enabled anything in the filesystem config options?"
[19:34] <rkeene> I would say it's a bug in Ceph, even without reading through it completely -- there should be no option (unless clearly marked) to enable data corruption.
[19:34] * kefu is now known as zz_kefu
[19:34] <rkeene> TheSov, That turned out not to be the case
[19:35] <TheSov> rkeene, oh? where do u see that?
[19:35] <rkeene> Root cause is a the log (which I assume is the journal?) being closed while still in use
[19:35] <rkeene> TheSov, The fact that the root cause is completely unrelated to that ?
[19:35] <burley> not the journal, the actual OSD log files
[19:36] <TheSov> according to the complaint they were logging to the OSD disks
[19:36] <TheSov> the first thing that pops into my head is that you should not be writing to a disk ceph uses for anything
[19:36] <darkfader> it's not incorrect usage if it dies from it
[19:37] <darkfader> seriously
[19:37] <rkeene> I don't think the issue is related to WHERE they were logging to -- just that the wrong file descriptor gets written to
[19:37] <TheSov> im looking 1 moment
[19:38] <rkeene> I think I understand the issue now -- a log file is closed but the file descriptor is still referenced by something wanting to write log entries, by the time the writer gets around to writing the fd has been re-used for OSD data, so when it *DOES* write it overwrites part of the OSD
[19:38] <TheSov> ok then that comes back full circle
[19:38] <rkeene> Fossil had a similar bug at one point
[19:38] <rkeene> It has no relationship to where the log file is being stored
[19:38] <TheSov> are you sure about that?
[19:39] <rkeene> Just that the file descriptor is written to after it's closed and re-opened
[19:39] <rkeene> Yes
[19:39] <rkeene> The file descriptor could reference a file on a v9fs filesystem hosted on the moon and it wouldn't be relevant
[19:39] <TheSov> so if u look at the most recent comments on that ticket
[19:40] <TheSov> i see what you are saying its the FD, but ceph doesnt write FD's
[19:40] * segutier (~segutier@172.56.20.151) has joined #ceph
[19:41] <rkeene> ceph-osd definitely writes to FDs, that's really the only facility you have for talking to files
[19:41] * Xa (~Shnaw@5NZAAFWCZ.tor-irc.dnsbl.oftc.net) Quit ()
[19:41] * allenmelon (~Bromine@9S0AAC2LB.tor-irc.dnsbl.oftc.net) has joined #ceph
[19:41] <TheSov> but i mean ceph doesnt actually create them, thats the OS
[19:41] * segutier (~segutier@172.56.20.151) Quit (Read error: Connection reset by peer)
[19:41] <rkeene> ceph-osd talks to the OS -- it *HAS* to
[19:41] <TheSov> right
[19:41] <TheSov> lets agree on that
[19:41] * wushudoin_ (~wushudoin@transit-86-181-132-209.redhat.com) has joined #ceph
[19:41] * brutuscat (~brutuscat@234.Red-79-151-98.dynamicIP.rima-tde.net) Quit (Remote host closed the connection)
[19:41] * TMM (~hp@178-84-46-106.dynamic.upc.nl) has joined #ceph
[19:42] <TheSov> so you are saying the FD was closed while in use?
[19:42] <rkeene> There's no such thing as "in use" here (well, there is but the kernel hides that concept)
[19:42] <rkeene> The FD was closed while there was something storing its value somewhere
[19:42] <TheSov> and ceph makes the call to close?
[19:43] <rkeene> Yes
[19:43] <rkeene> To rotate the log file
[19:43] * ircolle (~ircolle@mobile-166-173-060-136.mycingular.net) Quit (Ping timeout: 480 seconds)
[19:43] <rkeene> But another thread opens another file first
[19:43] <rkeene> And that FD gets assigned to that file
[19:43] <rkeene> And so the rotation of the log file gets a new FD
[19:43] <TheSov> so pardon my ignorance but shouldnt writes for that FD get stuck in queue while this happens?
[19:43] <rkeene> But whoever was referncing the old FD
[19:43] <rkeene> What ?
[19:44] <TheSov> im just thinking that if i want to make writes, and i need to move to a different file but use the same descriptor, shouldnt writes be held pending until that transfer is made?
[19:44] <rkeene> write(2) will succeed or fail -- whether they get stuck in a queue is irrelevant.
[19:45] <TheSov> ok i see what you are saying
[19:45] <rkeene> The thing is they didn't need to use the same file descriptor, when they open the next log file for log file rotation they just store the new descriptor and use it.
[19:45] <rkeene> But something in Ceph didn't get updated and was using the old descriptor (which refers to a different file than the log file)
[19:45] <TheSov> now i understand why its probably a ceph issue
[19:46] <TheSov> thanks
[19:46] <TheSov> so the question now is, why did that happen
[19:48] <rkeene> So the serialized sequence (keep in mind this is threaded) looks something like this: int logfd = open(osdlogfile, O_WRONLY | O_APPEND | O_CREAT, 0600); write(logfd, "my sweet log message\n", 21); close(logfd); int osdfd = open(osdfile, O_RDWR | O_CREAT, 0666); write(logfd, "more logs but wait, what ?\n", ...) ; logfd = open(logfile, ...); write(logfd, "Hey this one made it !\n", ...);
[19:48] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[19:48] <rkeene> TheSov, According to the patch it looks like they didn't put a mutex around the section of code that re-opens the log file, so writes can proceed while the re-open is happening
[19:49] * wushudoin (~wushudoin@38.140.108.2) Quit (Ping timeout: 480 seconds)
[19:49] <TheSov> ouch
[19:50] * wushudoin_ (~wushudoin@transit-86-181-132-209.redhat.com) Quit (Ping timeout: 480 seconds)
[19:50] <rkeene> "more logs but wait, what ?\n" gets written to the OSD file in this case, whereever it happens to be positioned, which is inconvienent for your data
[19:51] <rkeene> burley, Any of that sound right from your understanding ?
[19:52] <burley> that is partial, but accurate
[19:52] * ilken (ilk@2602:63:c2a2:af00:50c3:e80a:1a21:c3e2) has joined #ceph
[19:52] <rkeene> Partial ?
[19:52] <burley> as noted, I expect we'll be detailing all the moving parts -- but that does capture the major defect involved
[19:52] <TheSov> wait is that purposefully done that way?
[19:52] <rkeene> TheSov, No, it's a bug
[19:53] <TheSov> and its been fixed now?
[19:53] <burley> to workaround that issue for now, disable OSD logging
[19:53] <rkeene> I mean, it's not a bug of undefined behaviour but of non-determinism through multithreading
[19:54] <rkeene> (A "race condition" specifically -- you have to execute the right code exactly in the window between close() and open())
[19:54] <TheSov> well thats still a little disconcerting
[19:54] <rkeene> TheSov, I think a proposed patch is in the upstream source, but I don't think that it's been released
[19:54] <rkeene> So it's not the intended behaviour, but it is the behaviour of the code as it was written
[19:55] <TheSov> well i have a test cluster we have been using and i was about to goto prod
[19:55] <TheSov> and now i have to like re-evaluate that
[19:55] <rkeene> So it was "purposefully done that way" (I mean, whoever wrote that code did it on purpose) but didn't capture their full intention most likely -- they weren't attempting to leave a race condition in there
[19:56] <rkeene> Are you using OSD logging ?
[19:56] <TheSov> no
[19:56] <TheSov> but we may in prod
[19:56] <rkeene> Not if you want to be completely sure you don't hit this bug you won't :-D
[19:57] <TheSov> well it makes me wonder what similar bugs may exist
[19:57] <burley> you'll also want to disable log rotation
[19:57] <rkeene> There are probably other bugs, Ceph is large (120MB compiled here :-/)
[19:57] * epf (epf@155.254.32.82) has joined #ceph
[19:57] <TheSov> so disable logging and log rotation?
[19:58] <burley> debug-{ms,filestore,osd} and log rotation on the osd nodes
[19:58] <TheSov> by default logging is disabled correct? or is it really low level
[19:59] * wushudoin_ (~wushudoin@38.140.108.2) has joined #ceph
[19:59] <rkeene> The amount of logging only determines how often you get the chance to hit this race bug. If you have log rotation on at all, there's a chance the two pieces of code will align correctly.
[20:00] * TMM (~hp@178-84-46-106.dynamic.upc.nl) Quit (Remote host closed the connection)
[20:01] * haomaiwang (~haomaiwan@li817-120.members.linode.com) Quit (Remote host closed the connection)
[20:01] * kmARC (~kmARC@84-73-73-158.dclient.hispeed.ch) has joined #ceph
[20:01] <TheSov> so disable logging and disable logrotation and i am safe from THIS ONE particular bug?
[20:02] * haomaiwang (~haomaiwan@li817-120.members.linode.com) has joined #ceph
[20:02] <rkeene> Yes (probably just disabling log rotation is sufficient to protect you from this one particular bug AIUI)
[20:03] * nhm (~nhm@65-128-188-206.mpls.qwest.net) Quit (Ping timeout: 480 seconds)
[20:04] * TMM (~hp@178-84-46-106.dynamic.upc.nl) has joined #ceph
[20:08] * ivotron (~ivotron@eduroam-169-233-197-33.ucsc.edu) has joined #ceph
[20:11] * allenmelon (~Bromine@9S0AAC2LB.tor-irc.dnsbl.oftc.net) Quit ()
[20:11] * Schaap (~mog_@exit1.blackcatz.org) has joined #ceph
[20:11] * jluis (~joao@249.38.136.95.rev.vodafone.pt) Quit (Ping timeout: 480 seconds)
[20:15] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[20:15] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[20:15] * erice (~erice@c-76-120-53-165.hsd1.co.comcast.net) has joined #ceph
[20:16] * amatter (~oftc-webi@208.186.235.4) has joined #ceph
[20:16] * nsoffer (~nsoffer@nat-pool-tlv-t.redhat.com) Quit (Ping timeout: 480 seconds)
[20:19] <amatter> Hi all. I have a four host, 12 OSD cluster. 2 OSDs are out and objects are degraded but none are unfound. However when I attempt to get a file using the rados command from the cluster, it only hangs indefinitely. What can I check? ceph -s is here http://pastebin.com/DHKzreBU
[20:19] <amatter> 3 of 3 mons are working
[20:23] * Aemerson (~Icedove@aa2.linuxbox.com) has left #ceph
[20:23] * erice (~erice@c-76-120-53-165.hsd1.co.comcast.net) Quit (Ping timeout: 480 seconds)
[20:23] <doppelgrau> amatter: size=min_size?
[20:24] * rotbeard (~redbeard@2a02:908:df18:6480:76f0:6dff:fe3b:994d) Quit (Quit: Verlassend)
[20:26] <amatter> doppelgrau: thanks for the response. min size of the pool used is 2, there should be plenty of room for replica objects
[20:26] <doppelgrau> amatter: and size is also 2?
[20:27] <amatter> amatter: 256 pgs in the pool
[20:29] * rotbeard (~redbeard@2a02:908:df18:6480:6267:20ff:feb7:c20) has joined #ceph
[20:31] * xdeller (~xdeller@h195-91-128-218.ln.rinet.ru) Quit (Ping timeout: 480 seconds)
[20:38] * kmARC_ (~kmARC@84-73-73-158.dclient.hispeed.ch) has joined #ceph
[20:38] * kmARC (~kmARC@84-73-73-158.dclient.hispeed.ch) Quit (Quit: Leaving)
[20:38] * kmARC_ (~kmARC@84-73-73-158.dclient.hispeed.ch) Quit ()
[20:38] * kmARC (~kmARC@84-73-73-158.dclient.hispeed.ch) has joined #ceph
[20:40] * segutier (~segutier@172.56.42.105) has joined #ceph
[20:41] * Schaap (~mog_@5NZAAFWFX.tor-irc.dnsbl.oftc.net) Quit ()
[20:41] * PcJamesy (~dontron@23.95.43.73) has joined #ceph
[20:45] * jcsp (~jspray@82-71-16-249.dsl.in-addr.zen.co.uk) has joined #ceph
[20:50] * jclm (~jclm@ip24-253-98-109.lv.lv.cox.net) Quit (Quit: Leaving.)
[20:52] * MACscr|lappy (~MACscrlap@96.90.110.84) Quit (Ping timeout: 480 seconds)
[21:01] * haomaiwang (~haomaiwan@li817-120.members.linode.com) Quit (Remote host closed the connection)
[21:02] * haomaiwang (~haomaiwan@li817-120.members.linode.com) has joined #ceph
[21:04] * segutier (~segutier@172.56.42.105) Quit (Ping timeout: 480 seconds)
[21:08] * rendar (~I@host224-193-dynamic.18-79-r.retail.telecomitalia.it) Quit (Ping timeout: 480 seconds)
[21:10] * rendar (~I@host224-193-dynamic.18-79-r.retail.telecomitalia.it) has joined #ceph
[21:11] * PcJamesy (~dontron@7R2AADDZ3.tor-irc.dnsbl.oftc.net) Quit ()
[21:11] * AGaW (~Chaos_Lla@7R2AADD1L.tor-irc.dnsbl.oftc.net) has joined #ceph
[21:12] <TheSov> so do i want around 200 PG's per osd or 200 PG's per pool?
[21:12] * jclm (~jclm@172.56.30.191) has joined #ceph
[21:12] * ngoswami (~ngoswami@121.244.87.116) Quit (Quit: Leaving)
[21:14] * kutija_ (~kutija@daikatana.services.mint.rs) has joined #ceph
[21:16] * CAPSLOCK2000 (~oftc@2001:984:3be3:1::8) has joined #ceph
[21:17] * angdraug (~angdraug@12.164.168.117) has joined #ceph
[21:19] <rkeene> I use the smaller of: 2**entier(ceil(log($num_osds * 100 / 3)/log(2))) and 32 * $num_osds
[21:19] <rkeene> And the greater of: 64 and the above value
[21:19] * wicope (~wicope@0001fd8a.user.oftc.net) Quit (Remote host closed the connection)
[21:20] * kutija (~kutija@95.180.90.38) Quit (Ping timeout: 480 seconds)
[21:21] * nsoffer (~nsoffer@bzq-109-65-255-15.red.bezeqint.net) has joined #ceph
[21:22] * fdmanana (~fdmanana@bl13-153-23.dsl.telepac.pt) Quit (Ping timeout: 480 seconds)
[21:27] * Aid (c00cb806@107.161.19.53) has joined #ceph
[21:27] * angdraug (~angdraug@12.164.168.117) Quit (Quit: Leaving)
[21:27] <Aid> Hi... when writing to s3 via a proxy node to an Erasure Coded pool does the EC happen on the proxy node or the OSD?
[21:28] * Aid is now known as Aid2
[21:32] * xarses (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) has joined #ceph
[21:34] * derjohn_mob (~aj@fw.gkh-setu.de) Quit (Ping timeout: 480 seconds)
[21:36] * ircolle (~ircolle@mobile-166-173-060-136.mycingular.net) has joined #ceph
[21:36] * thomnico (~thomnico@2a01:e35:8b41:120:5194:ce76:9ebe:30) Quit (Ping timeout: 480 seconds)
[21:36] <m0zes> iirc, the client has to do the ec calculations.
[21:37] <Aid2> Thx :)
[21:38] * angdraug (~angdraug@12.164.168.117) has joined #ceph
[21:41] * AGaW (~Chaos_Lla@7R2AADD1L.tor-irc.dnsbl.oftc.net) Quit ()
[21:41] * Drezil (~rogst@tor00.telenet.unc.edu) has joined #ceph
[21:46] * dopesong (~dopesong@78-60-74-130.static.zebra.lt) Quit (Remote host closed the connection)
[21:46] * dneary (~dneary@50-204-53-85-static.hfc.comcastbusiness.net) has joined #ceph
[21:48] * angdraug (~angdraug@12.164.168.117) Quit (Quit: Leaving)
[21:48] * angdraug (~angdraug@12.164.168.117) has joined #ceph
[21:53] * CephFan1 (~textual@68-233-224-176.static.hvvc.us) has joined #ceph
[21:53] * vikhyat (~vumrao@49.248.203.182) Quit (Quit: Leaving)
[21:55] * nhm (~nhm@172.56.31.11) has joined #ceph
[21:55] * ChanServ sets mode +o nhm
[21:57] <TheSov> so that fix for the logging/race condition FD dealy, is that already in the latest ceph builds?
[21:57] <kutija_> m0zes (and others who talked to me this morning in order to get me through the mess)
[21:57] <kutija_> I found the solution
[21:58] <kutija_> so if you loose your partitions on Journal
[21:58] <kutija_> they are probably not lost except if SSD or whatever you use for journal is physically dead
[21:58] * snakamoto (~Adium@2600:1012:b01b:e9cc:d59a:c73b:18ca:8274) has joined #ceph
[21:58] <kutija_> in my case, GPT was damaged
[21:59] <kutija_> so after little work with gdisk I've managed to get them back
[21:59] <kutija_> and to start my OSD's without any issues
[22:00] <kutija_> here you can find how to fix GPT issues
[22:00] <kutija_> https://gist.github.com/jlee42/3062069
[22:00] <kutija_> other solution to the problem is this
[22:00] <kutija_> http://www.sebastien-han.fr/blog/2014/11/27/ceph-recover-osds-after-ssd-journal-failure/
[22:00] <m0zes> kutija_: did you find out how the gpt partition disappeared?
[22:00] <kutija_> no
[22:00] <kutija_> still not
[22:01] * haomaiwang (~haomaiwan@li817-120.members.linode.com) Quit (Remote host closed the connection)
[22:01] * xarses (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[22:02] * haomaiwang (~haomaiwan@li817-120.members.linode.com) has joined #ceph
[22:02] * angdraug (~angdraug@12.164.168.117) Quit (Quit: Leaving)
[22:02] <kutija_> actually I am not sure how to do that
[22:02] <kutija_> since all my logs are clean
[22:02] * angdraug (~angdraug@12.164.168.117) has joined #ceph
[22:02] * fdmanana (~fdmanana@bl13-153-23.dsl.telepac.pt) has joined #ceph
[22:04] <kutija_> but I am thrilled that my cluster is recovering
[22:04] <kutija_> and I will definitely increase my replication to 3 at the next storage upgrade
[22:05] * joao (~joao@249.38.136.95.rev.vodafone.pt) has joined #ceph
[22:05] * ChanServ sets mode +o joao
[22:11] * Drezil (~rogst@7R2AADD2Q.tor-irc.dnsbl.oftc.net) Quit ()
[22:11] * basicxman (~Kottizen@5NZAAFWMD.tor-irc.dnsbl.oftc.net) has joined #ceph
[22:16] * erice (~erice@c-76-120-53-165.hsd1.co.comcast.net) has joined #ceph
[22:18] * fdmanana (~fdmanana@bl13-153-23.dsl.telepac.pt) Quit (Ping timeout: 480 seconds)
[22:22] * dneary (~dneary@50-204-53-85-static.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[22:24] * erice (~erice@c-76-120-53-165.hsd1.co.comcast.net) Quit (Ping timeout: 480 seconds)
[22:31] * segutier (~segutier@172.56.16.253) has joined #ceph
[22:36] <burley> TheSov: No
[22:39] <rkeene> (This is one reason I like the Tcl-style apartment threading model -- this particular issue couldn't happen and races are much less common in event models versus threaded models since the overlap points are fairly well defined instead of just "anywhere")
[22:41] * basicxman (~Kottizen@5NZAAFWMD.tor-irc.dnsbl.oftc.net) Quit ()
[22:41] * tupper (~tcole@173.38.117.89) Quit (Quit: Leaving)
[22:45] * fdmanana (~fdmanana@bl13-153-23.dsl.telepac.pt) has joined #ceph
[22:46] * segutier (~segutier@172.56.16.253) Quit (Ping timeout: 480 seconds)
[22:48] * xarses (~xarses@12.164.168.117) has joined #ceph
[22:50] * segutier (~segutier@172.56.16.253) has joined #ceph
[22:51] * ivotron (~ivotron@eduroam-169-233-197-33.ucsc.edu) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[22:52] * brutuscat (~brutuscat@105.34.133.37.dynamic.jazztel.es) has joined #ceph
[23:01] * haomaiwang (~haomaiwan@li817-120.members.linode.com) Quit (Remote host closed the connection)
[23:01] * haomaiwang (~haomaiwan@li817-120.members.linode.com) has joined #ceph
[23:04] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit (Ping timeout: 480 seconds)
[23:07] <TheSov> burley, damn! when it that slated for release?
[23:08] * segutier (~segutier@172.56.16.253) Quit (Ping timeout: 480 seconds)
[23:08] <burley> TheSov: I am sure it will be released as soon as they have run it through all the normal release tests
[23:09] * beardo_ (~sma310@207-172-244-241.c3-0.atw-ubr5.atw.pa.cable.rcn.com) has joined #ceph
[23:11] * Salamander_ (~CobraKhan@tor.nohats.ca) has joined #ceph
[23:11] * badone_ (~brad@CPE-121-215-241-179.static.qld.bigpond.net.au) has joined #ceph
[23:13] * nsoffer (~nsoffer@bzq-109-65-255-15.red.bezeqint.net) Quit (Ping timeout: 480 seconds)
[23:14] * nsoffer (~nsoffer@109.65.255.15) has joined #ceph
[23:16] * ircolle (~ircolle@mobile-166-173-060-136.mycingular.net) Quit (Ping timeout: 480 seconds)
[23:19] * fdmanana (~fdmanana@bl13-153-23.dsl.telepac.pt) Quit (Ping timeout: 480 seconds)
[23:19] * DV_ (~veillard@2001:41d0:1:d478::1) Quit (Ping timeout: 480 seconds)
[23:20] * segutier (~segutier@172.56.14.76) has joined #ceph
[23:28] * ivotron (~ivotron@c-67-169-145-20.hsd1.ca.comcast.net) has joined #ceph
[23:41] * Salamander_ (~CobraKhan@7R2AADD6H.tor-irc.dnsbl.oftc.net) Quit ()
[23:41] * hassifa (~sardonyx@7R2AADD7W.tor-irc.dnsbl.oftc.net) has joined #ceph
[23:42] * DV (~veillard@2001:41d0:1:d478::1) has joined #ceph
[23:42] * qhartman (~qhartman@den.direwolfdigital.com) has joined #ceph
[23:43] * foxdalas (~maximpogo@79.165.44.31) has joined #ceph
[23:46] * linjan_ (~linjan@176.195.232.220) Quit (Ping timeout: 480 seconds)
[23:47] * brad_mssw (~brad@66.129.88.50) Quit (Quit: Leaving)
[23:49] * sleinen (~Adium@172.113.77.83.dynamic.wline.res.cust.swisscom.ch) has joined #ceph
[23:50] * sleinen1 (~Adium@2001:620:0:69::101) has joined #ceph
[23:55] * alfredodeza (~alfredode@198.206.133.89) Quit (Ping timeout: 480 seconds)
[23:56] * alfredodeza (~alfredode@198.206.133.89) has joined #ceph
[23:57] * sleinen (~Adium@172.113.77.83.dynamic.wline.res.cust.swisscom.ch) Quit (Ping timeout: 480 seconds)

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.