#ceph IRC Log

Index

IRC Log for 2015-02-17

Timestamps are in GMT/BST.

[0:01] * vakulkar (~vakulkar@nat-pool-rdu-u.redhat.com) Quit (Ping timeout: 480 seconds)
[0:03] * zack_dolby (~textual@pa3b3a1.tokynt01.ap.so-net.ne.jp) Quit (Quit: My MacBook has gone to sleep. ZZZzzz???)
[0:04] * vata (~vata@208.88.110.46) Quit (Quit: Leaving.)
[0:04] * oro (~oro@80-219-254-208.dclient.hispeed.ch) Quit (Ping timeout: 480 seconds)
[0:05] * gregmark (~Adium@68.87.42.115) Quit (Quit: Leaving.)
[0:09] * dgbaley27 (~matt@ucb-np1-206.colorado.edu) Quit (Quit: Leaving.)
[0:16] * tobiash (~quassel@host-88-217-137-244.customer.m-online.net) has joined #ceph
[0:20] * saltlake (~saltlake@pool-71-244-62-208.dllstx.fios.verizon.net) Quit (Ping timeout: 480 seconds)
[0:20] * tobiash_ (~quassel@mail.bmw-carit.de) Quit (Ping timeout: 480 seconds)
[0:20] * thb (~me@0001bd58.user.oftc.net) Quit (Ping timeout: 480 seconds)
[0:25] * joshd (~joshd@67-203-191-242.static-ip.telepacific.net) has joined #ceph
[0:34] * nitti (~nitti@c-66-41-30-224.hsd1.mn.comcast.net) Quit (Remote host closed the connection)
[0:41] * jdillaman (~jdillaman@pool-108-56-67-212.washdc.fios.verizon.net) Quit (Quit: jdillaman)
[0:43] * immesys (sid44615@id-44615.charlton.irccloud.com) has left #ceph
[0:44] * jdillaman (~jdillaman@pool-108-56-67-212.washdc.fios.verizon.net) has joined #ceph
[0:45] * jdillaman (~jdillaman@pool-108-56-67-212.washdc.fios.verizon.net) Quit ()
[0:50] * saltlake (~saltlake@pool-71-244-62-208.dllstx.fios.verizon.net) has joined #ceph
[0:53] * PerlStalker (~PerlStalk@2620:d3:8000:192::70) Quit (Quit: ...)
[0:57] * zack_dolby (~textual@nfmv008157.uqw.ppp.infoweb.ne.jp) has joined #ceph
[0:58] * togdon (~togdon@74.121.28.6) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[1:01] * dmsimard is now known as dmsimard_away
[1:03] * togdon (~togdon@74.121.28.6) has joined #ceph
[1:13] * joshd (~joshd@67-203-191-242.static-ip.telepacific.net) Quit (Quit: Leaving.)
[1:14] * qhartman (~qhartman@den.direwolfdigital.com) has joined #ceph
[1:16] <qhartman> I'm trying to create an erasure coded pool on Firefly, and get an error about not being able to find libec_jerasure.so when I run the command to do it. Any suggestions?
[1:17] <qhartman> full command and output: http://pastebin.com/MV3riqRv
[1:18] <qhartman> I've symlinked the files into the path requested and still no love, so I'm guessing this error is something of a red-herring.
[1:18] * jdillaman (~jdillaman@pool-108-56-67-212.washdc.fios.verizon.net) has joined #ceph
[1:20] <qhartman> I found references to an email thread where someone ran into this problem last year, but I get 404's when I try to actually visit the mail archives.
[1:22] <qhartman> aha, found the email archive, poking through the thread....
[1:25] <qhartman> hm, no resolution on that thread
[1:29] * saltlake (~saltlake@pool-71-244-62-208.dllstx.fios.verizon.net) Quit (Ping timeout: 480 seconds)
[1:29] * bandrus (~brian@198.23.71.75-static.reverse.softlayer.com) Quit (Quit: Leaving.)
[1:31] * bandrus (~brian@198.23.71.75-static.reverse.softlayer.com) has joined #ceph
[1:31] * togdon (~togdon@74.121.28.6) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[1:31] * togdon (~togdon@74.121.28.6) has joined #ceph
[1:32] * jdillaman (~jdillaman@pool-108-56-67-212.washdc.fios.verizon.net) Quit (Quit: jdillaman)
[1:37] * togdon (~togdon@74.121.28.6) Quit (Quit: Textual IRC Client: www.textualapp.com)
[1:50] * saltlake (~saltlake@pool-71-244-62-208.dllstx.fios.verizon.net) has joined #ceph
[1:50] * zack_dolby (~textual@nfmv008157.uqw.ppp.infoweb.ne.jp) Quit (Read error: Connection reset by peer)
[1:50] * zack_dolby (~textual@nfmv008157.uqw.ppp.infoweb.ne.jp) has joined #ceph
[1:55] * LeaChim (~LeaChim@host86-159-114-39.range86-159.btcentralplus.com) Quit (Ping timeout: 480 seconds)
[2:00] * oms101 (~oms101@p20030057EA08CF00EEF4BBFFFE0F7062.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[2:07] * vakulkar (~vakulkar@c-50-185-132-102.hsd1.ca.comcast.net) has joined #ceph
[2:07] * nitti (~nitti@c-66-41-30-224.hsd1.mn.comcast.net) has joined #ceph
[2:08] * oms101 (~oms101@p20030057EA07FE00EEF4BBFFFE0F7062.dip0.t-ipconnect.de) has joined #ceph
[2:09] * nhm (~nhm@65-128-165-174.mpls.qwest.net) Quit (Remote host closed the connection)
[2:09] * nhm (~nhm@65-128-165-174.mpls.qwest.net) has joined #ceph
[2:09] * ChanServ sets mode +o nhm
[2:11] * sudocat (~davidi@192.185.1.20) Quit (Ping timeout: 480 seconds)
[2:17] * danieagle (~Daniel@201-95-103-54.dsl.telesp.net.br) Quit (Quit: Obrigado por Tudo! :-) inte+ :-))
[2:19] * puffy (~puffy@216.207.42.129) Quit (Ping timeout: 480 seconds)
[2:19] * nitti (~nitti@c-66-41-30-224.hsd1.mn.comcast.net) Quit (Remote host closed the connection)
[2:19] * badone_ (~brad@CPE-121-215-241-179.static.qld.bigpond.net.au) has joined #ceph
[2:24] * nitti (~nitti@c-66-41-30-224.hsd1.mn.comcast.net) has joined #ceph
[2:25] * badone (~brad@CPE-121-215-241-179.static.qld.bigpond.net.au) Quit (Ping timeout: 480 seconds)
[2:32] * badone_ is now known as badone
[2:33] * nitti (~nitti@c-66-41-30-224.hsd1.mn.comcast.net) Quit (Remote host closed the connection)
[2:37] * saltlake (~saltlake@pool-71-244-62-208.dllstx.fios.verizon.net) Quit (Ping timeout: 480 seconds)
[2:47] * sudocat (~davidi@2601:e:2b80:9920:746a:8694:a2b6:c5ab) has joined #ceph
[2:51] * cholcombe973 (~chris@7208-76ef-ff1f-ed2f-329a-f002-3420-2062.6rd.ip6.sonic.net) Quit (Remote host closed the connection)
[2:52] * sudocat (~davidi@2601:e:2b80:9920:746a:8694:a2b6:c5ab) Quit (Quit: Leaving.)
[2:53] * sudocat (~davidi@2601:e:2b80:9920:224:d7ff:fe13:a040) has joined #ceph
[2:56] * avozza (~avozza@83.162.204.36) Quit (Remote host closed the connection)
[2:59] * georgem (~Adium@69-165-159-72.dsl.teksavvy.com) has joined #ceph
[2:59] * georgem (~Adium@69-165-159-72.dsl.teksavvy.com) has left #ceph
[3:00] * tobiash_ (~quassel@mail.bmw-carit.de) has joined #ceph
[3:00] * tobiash (~quassel@host-88-217-137-244.customer.m-online.net) Quit (Read error: Connection reset by peer)
[3:03] * zack_dol_ (~textual@pw126152000153.10.panda-world.ne.jp) has joined #ceph
[3:08] * zack_dolby (~textual@nfmv008157.uqw.ppp.infoweb.ne.jp) Quit (Ping timeout: 480 seconds)
[3:14] * vakulkar (~vakulkar@c-50-185-132-102.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[3:15] * sudocat (~davidi@2601:e:2b80:9920:224:d7ff:fe13:a040) Quit (Ping timeout: 480 seconds)
[3:21] * zack_dolby (~textual@nfmv008157.uqw.ppp.infoweb.ne.jp) has joined #ceph
[3:21] * zack_dol_ (~textual@pw126152000153.10.panda-world.ne.jp) Quit (Read error: Connection reset by peer)
[3:22] * clayb (~clayb@acehotel215.a.subnet.rcn.com) Quit (Ping timeout: 480 seconds)
[3:25] * dmsimard_away is now known as dmsimard
[3:25] * sw3_ (~oftc-webi@dyn-130-194-109-197.its.monash.edu.au) has joined #ceph
[3:27] * sw3_ (~oftc-webi@dyn-130-194-109-197.its.monash.edu.au) Quit ()
[3:29] * sw3 (sweaung@2400:6180:0:d0::66:100f) has joined #ceph
[3:31] * macjack (~Thunderbi@123.51.160.200) Quit (Quit: macjack)
[3:39] * macjack (~Thunderbi@123.51.160.200) has joined #ceph
[3:40] * shang (~ShangWu@175.41.48.77) has joined #ceph
[3:41] * cylee (~ubuntu@123.51.160.200) has joined #ceph
[3:42] * SamYaple_ is now known as SamYaple
[3:46] * cylee (~ubuntu@123.51.160.200) Quit ()
[3:47] * sudocat (~davidi@2601:e:2b80:9920:746a:8694:a2b6:c5ab) has joined #ceph
[4:09] * fam_away is now known as fam
[4:33] * cooldharma06 (~chatzilla@14.139.180.52) Quit (Quit: ChatZilla 0.9.91.1 [Iceweasel 21.0/20130515140136])
[4:40] * kanagaraj (~kanagaraj@121.244.87.117) has joined #ceph
[4:50] * rljohnsn (~rljohnsn@c-73-15-126-4.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[4:56] * reed (~reed@75-101-54-131.dsl.static.fusionbroadband.com) Quit (Quit: Ex-Chat)
[4:56] * vbellur (~vijay@122.167.110.132) has joined #ceph
[5:01] * zack_dol_ (~textual@nfmv001069076.uqw.ppp.infoweb.ne.jp) has joined #ceph
[5:03] * zack_dolby (~textual@nfmv008157.uqw.ppp.infoweb.ne.jp) Quit (Ping timeout: 480 seconds)
[5:04] * sjm (~sjm@pool-98-109-11-113.nwrknj.fios.verizon.net) has left #ceph
[5:10] * puffy (~puffy@50.185.218.255) has joined #ceph
[5:14] * shang (~ShangWu@175.41.48.77) Quit (Ping timeout: 480 seconds)
[5:17] * rljohnsn (~rljohnsn@c-73-15-126-4.hsd1.ca.comcast.net) has joined #ceph
[5:19] * Concubidated (~Adium@71.21.5.251) Quit (Quit: Leaving.)
[5:25] * puffy (~puffy@50.185.218.255) Quit (Quit: Leaving.)
[5:27] * Vacuum (~vovo@i59F79BB8.versanet.de) has joined #ceph
[5:30] * vbellur (~vijay@122.167.110.132) Quit (Ping timeout: 480 seconds)
[5:34] * nitti (~nitti@c-66-41-30-224.hsd1.mn.comcast.net) has joined #ceph
[5:34] * Vacuum_ (~vovo@88.130.216.36) Quit (Ping timeout: 480 seconds)
[5:41] * vbellur (~vijay@122.167.110.247) has joined #ceph
[5:42] * nitti (~nitti@c-66-41-30-224.hsd1.mn.comcast.net) Quit (Ping timeout: 480 seconds)
[5:49] * puffy (~puffy@50.185.218.255) has joined #ceph
[5:53] * puffy (~puffy@50.185.218.255) Quit ()
[6:02] * masterpe_ (~masterpe@2a01:670:400::43) Quit (Ping timeout: 480 seconds)
[6:03] * dmsimard is now known as dmsimard_away
[6:06] * masterpe (~masterpe@2a01:670:400::43) has joined #ceph
[6:08] <tserong> i feel faintly embarrassed to be asking this, but is it sage "wheel", or sage "while", when pronounced out loud?
[6:15] * rljohnsn (~rljohnsn@c-73-15-126-4.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[6:15] * vbellur (~vijay@122.167.110.247) Quit (Read error: Connection reset by peer)
[6:15] * lalatenduM (~lalatendu@121.244.87.117) has joined #ceph
[6:18] * kefu (~kefu@114.92.100.153) has joined #ceph
[6:22] * rdas (~rdas@121.244.87.116) has joined #ceph
[6:30] * vbellur (~vijay@122.167.246.177) has joined #ceph
[6:33] * karnan (~karnan@106.51.234.138) has joined #ceph
[6:34] * sudocat (~davidi@2601:e:2b80:9920:746a:8694:a2b6:c5ab) Quit (Quit: Leaving.)
[6:34] * sudocat (~davidi@73.166.99.97) has joined #ceph
[6:43] * bandrus (~brian@198.23.71.75-static.reverse.softlayer.com) Quit (Quit: Leaving.)
[6:44] * joshd (~joshd@8.25.222.10) has joined #ceph
[6:45] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[6:46] * sudocat (~davidi@73.166.99.97) Quit (Ping timeout: 480 seconds)
[6:47] * cooldharma06 (~chatzilla@14.139.180.52) has joined #ceph
[6:55] * KevinPerks (~Adium@cpe-071-071-026-213.triad.res.rr.com) Quit (Quit: Leaving.)
[6:55] * derjohn_mob (~aj@tmo-109-21.customers.d1-online.com) Quit (Ping timeout: 480 seconds)
[6:59] * nwf (~nwf@00018577.user.oftc.net) has joined #ceph
[7:05] * mookins (~mookins@induct3.lnk.telstra.net) has joined #ceph
[7:07] * fam is now known as fam_away
[7:11] * overclk (~overclk@121.244.87.117) has joined #ceph
[7:12] * vbellur (~vijay@122.167.246.177) Quit (Ping timeout: 480 seconds)
[7:15] * rljohnsn (~rljohnsn@c-73-15-126-4.hsd1.ca.comcast.net) has joined #ceph
[7:17] * fam_away is now known as fam
[7:18] * zack_dol_ (~textual@nfmv001069076.uqw.ppp.infoweb.ne.jp) Quit (Quit: My MacBook has gone to sleep. ZZZzzz???)
[7:19] * wschulze (~wschulze@cpe-69-206-251-158.nyc.res.rr.com) Quit (Quit: Leaving.)
[7:23] * vbellur (~vijay@122.178.231.105) has joined #ceph
[7:23] * kefu (~kefu@114.92.100.153) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[7:25] * kefu (~kefu@114.92.100.153) has joined #ceph
[7:29] * kefu (~kefu@114.92.100.153) Quit (Max SendQ exceeded)
[7:31] * avozza (~avozza@83.162.204.36) has joined #ceph
[7:33] * kefu (~kefu@114.92.100.153) has joined #ceph
[7:42] * derjohn_mob (~aj@88.128.80.178) has joined #ceph
[7:47] * kefu (~kefu@114.92.100.153) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[7:54] * nardial (~ls@ipservice-092-209-178-132.092.209.pools.vodafone-ip.de) has joined #ceph
[7:57] * alram (~alram@ppp-seco11pa2-46-193-132-162.wb.wifirst.net) has joined #ceph
[8:02] * dgbaley27 (~matt@c-67-176-93-83.hsd1.co.comcast.net) has joined #ceph
[8:07] * zack_dolby (~textual@219.117.239.161.static.zoot.jp) has joined #ceph
[8:07] * nhm (~nhm@65-128-165-174.mpls.qwest.net) Quit (Remote host closed the connection)
[8:08] * nhm (~nhm@65-128-165-174.mpls.qwest.net) has joined #ceph
[8:08] * ChanServ sets mode +o nhm
[8:17] * linjan (~linjan@195.110.41.9) has joined #ceph
[8:18] * rotbeard (~redbeard@2a02:908:df10:d300:76f0:6dff:fe3b:994d) Quit (Quit: Verlassend)
[8:21] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit (Ping timeout: 480 seconds)
[8:27] * swami1 (~swami@49.32.0.134) has joined #ceph
[8:27] * oro (~oro@80-219-254-208.dclient.hispeed.ch) has joined #ceph
[8:30] * sputnik13 (~sputnik13@c-73-193-97-20.hsd1.wa.comcast.net) has joined #ceph
[8:34] * vbellur (~vijay@122.178.231.105) Quit (Ping timeout: 480 seconds)
[8:34] * derjohn_mob (~aj@88.128.80.178) Quit (Ping timeout: 480 seconds)
[8:36] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) has joined #ceph
[8:41] * nils_ (~nils@doomstreet.collins.kg) has joined #ceph
[8:44] * sleinen1 (~Adium@2001:620:0:82::101) Quit (Read error: Connection reset by peer)
[8:45] * alram (~alram@ppp-seco11pa2-46-193-132-162.wb.wifirst.net) Quit (Ping timeout: 480 seconds)
[8:46] * joshd (~joshd@8.25.222.10) Quit (Quit: Leaving.)
[8:48] * kefu (~kefu@114.92.100.153) has joined #ceph
[8:50] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) Quit (Remote host closed the connection)
[8:52] * dgbaley27 (~matt@c-67-176-93-83.hsd1.co.comcast.net) Quit (Remote host closed the connection)
[8:54] * thb (~me@port-93387.pppoe.wtnet.de) has joined #ceph
[8:56] * sleinen (~Adium@2001:620:0:2d:7ed1:c3ff:fedc:3223) has joined #ceph
[8:56] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) has joined #ceph
[9:04] * ngoswami (~ngoswami@121.244.87.116) has joined #ceph
[9:05] * analbeard (~shw@support.memset.com) has joined #ceph
[9:07] * chasmo77 (~chas77@158.183-62-69.ftth.swbr.surewest.net) has joined #ceph
[9:08] * nhm (~nhm@65-128-165-174.mpls.qwest.net) Quit (Ping timeout: 480 seconds)
[9:08] * kawa2014 (~kawa@89.184.114.246) has joined #ceph
[9:08] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) Quit (Remote host closed the connection)
[9:10] * derjohn_mob (~aj@94.119.1.11) has joined #ceph
[9:11] * oro (~oro@80-219-254-208.dclient.hispeed.ch) Quit (Ping timeout: 480 seconds)
[9:11] * swami1 (~swami@49.32.0.134) has left #ceph
[9:11] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) has joined #ceph
[9:14] * nhm (~nhm@65-128-165-174.mpls.qwest.net) has joined #ceph
[9:14] * ChanServ sets mode +o nhm
[9:15] * rossmartyn04 (~rnm@support.memset.com) has joined #ceph
[9:16] <rossmartyn04> Hi, Is anyone free to take a look at a ceph-osd issue I am having? Following a power cut last week, I have three OSD's that will not start. They appeared to be scrubbing during the power cut, and are now logging the following
[9:16] <rossmartyn04> 2015-02-13 13:52:00.080038 7fe32be9d900 -1 osd/PG.cc: In function 'static epoch_t PG::peek_map_epoch(ObjectStore*, coll_t, hobject_t&, ceph::bufferlist*)' thread 7fe32be9d900 time 2015-02-13 13:52:00.078366
[9:16] <rossmartyn04> osd/PG.cc: 2683: FAILED assert(values.size() == 1)
[9:16] <rossmartyn04> ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578)
[9:16] <rossmartyn04> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xb8231b]
[9:16] <rossmartyn04> 2: (PG::peek_map_epoch(ObjectStore*, coll_t, hobject_t&, ceph::buffer::list*)+0x508) [0x7bb048]
[9:16] <rossmartyn04> 3: (OSD::load_pgs()+0xe4f) [0x6aa0ef]
[9:16] <rossmartyn04> 4: (OSD::init()+0x71f) [0x6abf5f]
[9:16] <rossmartyn04> 5: (main()+0x252c) [0x638cfc]
[9:16] <rossmartyn04> 6: (__libc_start_main()+0xf5) [0x7fe328fd9ec5]
[9:16] <rossmartyn04> 7: /usr/bin/ceph-osd() [0x651027]
[9:16] <rossmartyn04> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
[9:18] * Be-El (~quassel@fb08-bcf-pc01.computational.bio.uni-giessen.de) has joined #ceph
[9:18] <Be-El> hi
[9:19] * i_m (~ivan.miro@deibp9eh1--blueice1n2.emea.ibm.com) has joined #ceph
[9:20] * nils_ (~nils@doomstreet.collins.kg) Quit (Quit: Leaving)
[9:22] * brutusca_ (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) has joined #ceph
[9:22] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) Quit (Remote host closed the connection)
[9:25] * Hell_Fire (~hellfire@123-243-155-184.static.tpgi.com.au) Quit (Quit: Konversation terminated!)
[9:28] * alram (~alram@LAubervilliers-656-1-17-4.w217-128.abo.wanadoo.fr) has joined #ceph
[9:29] * rljohnsn (~rljohnsn@c-73-15-126-4.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[9:29] * TMM (~hp@178-84-46-106.dynamic.upc.nl) Quit (Quit: Ex-Chat)
[9:31] * jtang (~jtang@109.255.42.21) has joined #ceph
[9:35] * brutusca_ (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) Quit (Read error: Connection reset by peer)
[9:36] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) has joined #ceph
[9:36] * nils_ (~nils@doomstreet.collins.kg) has joined #ceph
[9:36] * fsimonce (~simon@host217-37-dynamic.30-79-r.retail.telecomitalia.it) has joined #ceph
[9:37] * rljohnsn (~rljohnsn@c-73-15-126-4.hsd1.ca.comcast.net) has joined #ceph
[9:38] * karnan (~karnan@106.51.234.138) Quit (Ping timeout: 480 seconds)
[9:42] <badone> rossmartyn04: http://tracker.ceph.com/issues/4855
[9:45] * jordanP (~jordan@213.215.2.194) has joined #ceph
[9:45] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) Quit (Read error: Connection reset by peer)
[9:48] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) has joined #ceph
[9:48] * rljohnsn (~rljohnsn@c-73-15-126-4.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[9:51] * nils_ (~nils@doomstreet.collins.kg) Quit (Quit: Leaving)
[9:51] * nils_ (~nils@doomstreet.collins.kg) has joined #ceph
[9:52] <rossmartyn04> Hi, Thanks for the information, I did catch that link. Though it does not look like a very simular situation. Not sure where to go from here!
[9:53] <rossmartyn04> Its a pre-production stack, but would be nice to know the fix!
[9:54] <rossmartyn04> We tried an XFS repair on one of the disks and we are now faced with the followibng :
[9:54] <rossmartyn04> 2015-02-17 08:46:50.704663 7f53feb11900 1 filestore(/var/lib/ceph/osd/ceph-8) disabling 'filestore replica fadvise' due to known issues with fadvise(DONTNEED) on xfs
[9:54] <rossmartyn04> 2015-02-17 08:46:50.707630 7f53feb11900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-8) detect_features: FIEMAP ioctl is supported and appears to work
[9:54] <rossmartyn04> 2015-02-17 08:46:50.707640 7f53feb11900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-8) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
[9:54] <rossmartyn04> 2015-02-17 08:46:50.708263 7f53feb11900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-8) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
[9:54] <rossmartyn04> 2015-02-17 08:46:50.708314 7f53feb11900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-8) detect_feature: extsize is disabled by conf
[9:54] <rossmartyn04> 2015-02-17 08:46:50.708924 7f53feb11900 -1 filestore(/var/lib/ceph/osd/ceph-8) Error initializing leveldb : Corruption: 7 missing files; e.g.: /var/lib/ceph/osd/ceph-8/current/omap/101816.ldb
[9:55] <badone> rossmartyn04: I'd probably just delete/remove the osds and add them back again
[9:55] <rossmartyn04> OK, we unfortunately have some PG's that are on two of the dead disks
[9:55] <badone> rossmartyn04: note that any data on the disks will be lost but it should have been replicated by now anyway i would imagine
[9:56] <badone> rossmartyn04: only on two of the dead disks?
[9:56] <rossmartyn04> OK 649 PG's as stale+active+clean
[9:56] <rossmartyn04> but not degraded. So maybe we are ok?
[9:57] <rossmartyn04> osdmap e31100: 14 osds: 11 up, 11 in
[9:57] <rossmartyn04> pgmap v1851023: 8064 pgs, 6 pools, 828 GB data, 438 kobjects
[9:57] <rossmartyn04> 1619 GB used, 13050 GB / 14670 GB avail
[9:57] <rossmartyn04> 649 stale+active+clean
[9:57] <rossmartyn04> 7415 active+clean
[9:57] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) Quit (Read error: Connection reset by peer)
[9:57] <badone> rossmartyn04: investigate the pgs involved. Start with a pg dump
[9:58] <rossmartyn04> Here is an example of three of them
[9:58] <rossmartyn04> 12.fd 0 0 0 0 0 0 0 0 stale+active+clean 2015-02-12 13:55:49.088237 0'0 22577:25725 [9,7] 9 [9,7] 9 0'0 2015-02-12 13:55:49.088208 0'0 2015-02-09 13:55:07.605457
[9:58] <rossmartyn04> 0.75f 0 0 0 0 0 0 0 0 stale+active+clean 2015-02-12 15:13:31.405104 0'0 22577:266 [7,8] 7 [7,8] 7 0'0 2015-02-12 15:13:31.405075 0'0 2015-02-12 15:13:31.405075
[9:58] <rossmartyn04> 12.fc 0 0 0 0 0 0 0 0 stale+active+clean 2015-02-12 13:46:58.837248 0'0 22577:2040 [7,9] 7 [7,9] 7 0'0 2015-02-12 13:46:58.837215 0'0 2015-02-09 13:46:10.755339
[9:58] <rossmartyn04> disks 7/8/9 are 'DOWN'
[9:58] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) has joined #ceph
[9:58] * greatmane (~greatmane@124.188.109.165) has joined #ceph
[9:58] <rossmartyn04> I was under the impression that this meant those blocks are only stored on 7/8
[9:59] <rossmartyn04> or 9/7
[10:01] <badone> ceph pg 12.fd query
[10:01] <badone> that should give you some more information
[10:02] <rossmartyn04> unfortunately not! :~# ceph pg 12.fd query
[10:02] <rossmartyn04> Error ENOENT: i don't have pgid 12.fd
[10:02] <rossmartyn04> same with the others
[10:03] <- *greatmane* help
[10:04] * greatmane (~greatmane@124.188.109.165) Quit ()
[10:04] * greatmane (~greatmane@124.188.109.165) has joined #ceph
[10:04] <badone> rossmartyn04: you should try not to have all osds in the same failure domain
[10:05] * TMM (~hp@sams-office-nat.tomtomgroup.com) has joined #ceph
[10:05] <rossmartyn04> Sure, as I said its testing/dev really. But noted, we will have more resilience built into our future stack
[10:06] <badone> rossmartyn04: sure
[10:06] * greatmane (~greatmane@124.188.109.165) Quit (Remote host closed the connection)
[10:07] * Nacer (~Nacer@252-87-190-213.intermediasud.com) has joined #ceph
[10:07] <rossmartyn04> OK, so am I to assume stuck stale basically means lost?
[10:09] <badone> rossmartyn04: ceph pg dump_stuck stale
[10:09] <badone> stale - The placement group status has not been updated by a ceph-osd, indicating that all nodes storing this placement group may be down.
[10:19] * Dasher (~oftc-webi@46.218.69.130) has joined #ceph
[10:19] * zack_dolby (~textual@219.117.239.161.static.zoot.jp) Quit (Quit: My MacBook has gone to sleep. ZZZzzz???)
[10:21] <badone> rossmartyn04: This is a bit beyond me at this stage. There may be someone with some thoughts but I don't know how to go about recovering something like that
[10:21] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) Quit (Read error: Connection reset by peer)
[10:22] * ninkotech (~duplo@static-84-242-87-186.net.upcbroadband.cz) has joined #ceph
[10:22] * karnan (~karnan@106.51.234.138) has joined #ceph
[10:23] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) has joined #ceph
[10:24] * shaunm (~shaunm@nat-pool-brq-t.redhat.com) has joined #ceph
[10:24] * shaunm (~shaunm@nat-pool-brq-t.redhat.com) Quit (Read error: Connection reset by peer)
[10:24] * shaunm (~shaunm@213.175.37.10) has joined #ceph
[10:26] * avozza (~avozza@83.162.204.36) Quit (Remote host closed the connection)
[10:26] * greatmane (~greatmane@124.188.109.165) has joined #ceph
[10:31] * nhm (~nhm@65-128-165-174.mpls.qwest.net) Quit (Ping timeout: 480 seconds)
[10:34] <rossmartyn04> No problem dude, thanks very much for your help
[10:34] <rossmartyn04> Ill be a bit cleverer with my next cluster!
[10:36] * Hell_Fire (~hellfire@123-243-155-184.static.tpgi.com.au) has joined #ceph
[10:36] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) Quit (Remote host closed the connection)
[10:40] * nhm (~nhm@65-128-165-174.mpls.qwest.net) has joined #ceph
[10:40] * ChanServ sets mode +o nhm
[10:41] * Sysadmin88 (~IceChat77@94.12.240.104) Quit (Quit: The early bird may get the worm, but the second mouse gets the cheese)
[10:42] * greatmane (~greatmane@124.188.109.165) Quit (Remote host closed the connection)
[10:44] * oro (~oro@2001:620:20:16:4142:225c:9779:f39c) has joined #ceph
[10:45] * greatmane (~greatmane@124.188.109.165) has joined #ceph
[10:56] * greatmane (~greatmane@124.188.109.165) Quit (Remote host closed the connection)
[10:57] * avozza (~avozza@nat-pool-ams-t.redhat.com) has joined #ceph
[10:58] * greatmane (~greatmane@124.188.109.165) has joined #ceph
[11:00] * avozza (~avozza@nat-pool-ams-t.redhat.com) Quit (Remote host closed the connection)
[11:02] * oro (~oro@2001:620:20:16:4142:225c:9779:f39c) Quit (Remote host closed the connection)
[11:03] * oro (~oro@2001:620:20:64:9858:9d3:b3e9:fc05) has joined #ceph
[11:05] * dgurtner (~dgurtner@217-162-119-191.dynamic.hispeed.ch) has joined #ceph
[11:05] * greatmane (~greatmane@124.188.109.165) Quit ()
[11:11] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) has joined #ceph
[11:15] * greatmane (~greatmane@124.188.109.165) has joined #ceph
[11:20] * brutusca_ (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) has joined #ceph
[11:20] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) Quit (Read error: Connection reset by peer)
[11:20] * danieagle (~Daniel@201-95-103-54.dsl.telesp.net.br) has joined #ceph
[11:23] * sugoruyo (~sug_@00014f5c.user.oftc.net) has joined #ceph
[11:24] * kapil (~ksharma@2620:113:80c0:5::2222) Quit (Quit: Leaving)
[11:25] <nils_> anyone try hybrid hdd/ssd for OSD? I think there is one which shows up as 2 devices to the host
[11:26] * avozza (~avozza@nat-pool-ams-t.redhat.com) has joined #ceph
[11:30] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) has joined #ceph
[11:30] * brutusca_ (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) Quit (Read error: Connection reset by peer)
[11:36] * Kioob`Taff (~plug-oliv@2a01:e35:2e8a:1e0::42:10) Quit (Quit: Leaving.)
[11:36] * Kioob`Taff (~plug-oliv@2a01:e35:2e8a:1e0::42:10) has joined #ceph
[11:38] <badone> rossmartyn04: you are most welcome
[11:42] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) Quit (Ping timeout: 480 seconds)
[11:44] * greatmane (~greatmane@124.188.109.165) Quit ()
[11:49] * fam is now known as fam_away
[11:59] * kefu (~kefu@114.92.100.153) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[12:03] * oro (~oro@2001:620:20:64:9858:9d3:b3e9:fc05) Quit (Ping timeout: 480 seconds)
[12:10] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) has joined #ceph
[12:11] * badone (~brad@CPE-121-215-241-179.static.qld.bigpond.net.au) Quit (Ping timeout: 480 seconds)
[12:23] * dmsimard_away (~dmsimard@198.72.123.202) Quit (Ping timeout: 480 seconds)
[12:27] * brutusca_ (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) has joined #ceph
[12:27] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) Quit (Read error: Connection reset by peer)
[12:27] * bitserker (~toni@63.pool85-52-240.static.orange.es) has joined #ceph
[12:27] * bitserker (~toni@63.pool85-52-240.static.orange.es) Quit (Remote host closed the connection)
[12:30] * dmsimard_away (~dmsimard@198.72.123.202) has joined #ceph
[12:30] * dmsimard_away is now known as dmsimard
[12:31] * kefu (~kefu@114.92.100.153) has joined #ceph
[12:39] * nhm (~nhm@65-128-165-174.mpls.qwest.net) Quit (Remote host closed the connection)
[12:39] * brutusca_ (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) Quit (Read error: Connection reset by peer)
[12:40] * nhm (~nhm@65-128-165-174.mpls.qwest.net) has joined #ceph
[12:40] * ChanServ sets mode +o nhm
[12:41] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) has joined #ceph
[12:45] * aarcane (~aarcane@99-42-64-118.lightspeed.irvnca.sbcglobal.net) Quit (Read error: Connection reset by peer)
[12:45] * alram (~alram@LAubervilliers-656-1-17-4.w217-128.abo.wanadoo.fr) Quit (Ping timeout: 480 seconds)
[12:51] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) Quit (Read error: Connection reset by peer)
[12:51] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) has joined #ceph
[12:51] <Be-El> does changing the pool attr for cephfs file also trigger a move of the file's content to a different pool?
[13:01] <jcsp> Be-Al: recent thread on that topic: http://www.spinics.net/lists/ceph-users/msg15395.html
[13:02] <jcsp> summary: you can't change the layout once a file has data in it, except you could, but you shouldn't have been able to, and that'll be fixed in the next version.
[13:02] <jcsp> so no, we don't do any data movement.
[13:05] <Be-El> jcsp: thx. does moving a file from one directory to another trigger an actual copy of the file content if both directories have different pools associated with them?
[13:06] <Be-El> or is moving a 'lazy' operation that just changes the underlying metadata?
[13:06] <jcsp> the file will keep its layout when you move it between folders
[13:06] <jcsp> the directory's layout only applies to newly created files
[13:06] <Be-El> ok, so i'll simple copy the later and delete the old files afterwards
[13:06] <Be-El> eh...data even
[13:08] * kanagaraj (~kanagaraj@121.244.87.117) Quit (Quit: Leaving)
[13:08] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) Quit (Read error: Connection reset by peer)
[13:10] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) has joined #ceph
[13:10] * TMM (~hp@sams-office-nat.tomtomgroup.com) Quit (Quit: Ex-Chat)
[13:13] * avozza (~avozza@nat-pool-ams-t.redhat.com) Quit (Ping timeout: 480 seconds)
[13:23] * brutusca_ (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) has joined #ceph
[13:23] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) Quit (Read error: Connection reset by peer)
[13:35] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) has joined #ceph
[13:35] * brutusca_ (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) Quit (Read error: Connection reset by peer)
[13:35] * shaunm (~shaunm@213.175.37.10) Quit (Ping timeout: 480 seconds)
[13:36] * oro (~oro@2001:620:20:16:9858:9d3:b3e9:fc05) has joined #ceph
[13:38] <bd> [2932804.674599] lost page write due to I/O error on rbd1
[13:39] <bd> looks like OOM
[13:43] <bd> fwiw http://paste.debian.net/hidden/d1cf4c6d/
[13:51] * overclk (~overclk@121.244.87.117) Quit (Quit: Leaving)
[13:51] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) Quit (Read error: Connection reset by peer)
[13:52] * alram (~alram@LAubervilliers-656-1-17-4.w217-128.abo.wanadoo.fr) has joined #ceph
[13:53] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) has joined #ceph
[14:00] * linjan (~linjan@195.110.41.9) Quit (Ping timeout: 480 seconds)
[14:05] * avozza (~avozza@62.140.132.226) has joined #ceph
[14:07] * avozza (~avozza@62.140.132.226) Quit (Remote host closed the connection)
[14:09] * linjan (~linjan@176.195.18.148) has joined #ceph
[14:09] * KevinPerks (~Adium@cpe-071-071-026-213.triad.res.rr.com) has joined #ceph
[14:10] * fdmanana__ (~fdmanana@bl4-182-212.dsl.telepac.pt) Quit (Quit: Leaving)
[14:14] * kefu (~kefu@114.92.100.153) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[14:15] * avozza (~avozza@62.140.132.226) has joined #ceph
[14:15] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) Quit (Read error: Connection reset by peer)
[14:17] * kefu (~kefu@114.92.100.153) has joined #ceph
[14:19] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) has joined #ceph
[14:22] * wschulze (~wschulze@cpe-69-206-251-158.nyc.res.rr.com) has joined #ceph
[14:22] * vbellur (~vijay@122.167.79.204) has joined #ceph
[14:23] * t0rn (~ssullivan@2607:fad0:32:a02:d227:88ff:fe02:9896) has joined #ceph
[14:25] * kefu (~kefu@114.92.100.153) Quit (Max SendQ exceeded)
[14:36] * bjornar (~bjornar@ns3.uniweb.no) Quit (Remote host closed the connection)
[14:39] * oro_ (~oro@2001:620:20:16:9858:9d3:b3e9:fc05) has joined #ceph
[14:40] * gregmark (~Adium@68.87.42.115) has joined #ceph
[14:45] * zack_dolby (~textual@pa3b3a1.tokynt01.ap.so-net.ne.jp) has joined #ceph
[14:45] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) Quit (Read error: Connection reset by peer)
[14:45] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) has joined #ceph
[14:57] * fdmanana (~fdmanana@bl4-182-212.dsl.telepac.pt) has joined #ceph
[14:59] * TMM (~hp@sams-office-nat.tomtomgroup.com) has joined #ceph
[15:04] * avozza (~avozza@62.140.132.226) Quit (Remote host closed the connection)
[15:05] * lalatenduM (~lalatendu@121.244.87.117) Quit (Quit: Leaving)
[15:07] * nardial (~ls@ipservice-092-209-178-132.092.209.pools.vodafone-ip.de) Quit (Quit: Leaving)
[15:07] * nitti (~nitti@162.222.47.218) has joined #ceph
[15:10] * shaunm (~shaunm@nat-pool-brq-t.redhat.com) has joined #ceph
[15:17] * brad_mssw (~brad@66.129.88.50) has joined #ceph
[15:17] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) Quit (Read error: Connection reset by peer)
[15:18] * rdas (~rdas@121.244.87.116) Quit (Quit: Leaving)
[15:18] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) has joined #ceph
[15:21] <sage> tserong: "while" (not "wheel")
[15:21] * jrankin (~jrankin@d53-64-170-236.nap.wideopenwest.com) has joined #ceph
[15:21] * tupper (~tcole@rtp-isp-nat1.cisco.com) Quit (Quit: Leaving)
[15:24] * rowie (~roche@2a01:e34:ec06:41f0::2) has joined #ceph
[15:30] * karnan (~karnan@106.51.234.138) Quit (Remote host closed the connection)
[15:32] * brutuscat (~brutuscat@73.Red-81-38-218.dynamicIP.rima-tde.net) Quit (Remote host closed the connection)
[15:35] * morse_ (~morse@supercomputing.univpm.it) Quit (Ping timeout: 480 seconds)
[15:37] * rljohnsn (~rljohnsn@c-73-15-126-4.hsd1.ca.comcast.net) has joined #ceph
[15:41] * morse (~morse@supercomputing.univpm.it) has joined #ceph
[15:48] * gregmark (~Adium@68.87.42.115) Quit (Quit: Leaving.)
[15:48] * sjm (~sjm@pool-98-109-11-113.nwrknj.fios.verizon.net) has joined #ceph
[15:48] * tupper_ (~tcole@rtp-isp-nat-pool1-1.cisco.com) Quit (Read error: Connection reset by peer)
[15:55] * rljohnsn (~rljohnsn@c-73-15-126-4.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[15:56] * avozza (~avozza@83.162.204.36) has joined #ceph
[15:57] * jcsp1 (~Adium@82-71-16-249.dsl.in-addr.zen.co.uk) has joined #ceph
[15:58] * dyasny (~dyasny@173.231.115.58) has joined #ceph
[15:59] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) has joined #ceph
[16:02] * PerlStalker (~PerlStalk@2620:d3:8000:192::70) has joined #ceph
[16:02] * tserong (~tserong@203-173-33-52.dyn.iinet.net.au) Quit (Ping timeout: 480 seconds)
[16:02] * jcsp (~Adium@0001bf3a.user.oftc.net) Quit (Ping timeout: 480 seconds)
[16:05] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) Quit (Read error: Connection reset by peer)
[16:05] * tupper_ (~tcole@108-83-203-37.lightspeed.rlghnc.sbcglobal.net) has joined #ceph
[16:06] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) has joined #ceph
[16:09] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) Quit (Read error: Connection reset by peer)
[16:09] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) has joined #ceph
[16:11] * jcsp (~Adium@82-71-16-249.dsl.in-addr.zen.co.uk) has joined #ceph
[16:16] * tserong (~tserong@203-173-33-52.dyn.iinet.net.au) has joined #ceph
[16:16] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) Quit (Read error: Connection reset by peer)
[16:16] * jcsp1 (~Adium@82-71-16-249.dsl.in-addr.zen.co.uk) Quit (Ping timeout: 480 seconds)
[16:18] * marrusl (~mark@cpe-24-90-46-248.nyc.res.rr.com) Quit (Remote host closed the connection)
[16:19] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) has joined #ceph
[16:20] * saltlake (~saltlake@12.250.199.170) has joined #ceph
[16:20] * marrusl (~mark@cpe-24-90-46-248.nyc.res.rr.com) has joined #ceph
[16:27] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) Quit (Read error: Connection reset by peer)
[16:27] * linuxkidd (~linuxkidd@73.sub-70-210-193.myvzw.com) has joined #ceph
[16:28] * xarses (~andreww@c-76-126-112-92.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[16:29] * debian112 (~bcolbert@24.126.201.64) has joined #ceph
[16:30] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) has joined #ceph
[16:35] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) Quit (Read error: Connection reset by peer)
[16:35] * cok (~chk@2a02:2350:18:1010:443f:556f:489e:ddbc) has joined #ceph
[16:36] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) has joined #ceph
[16:42] * cok (~chk@2a02:2350:18:1010:443f:556f:489e:ddbc) Quit (Quit: Leaving.)
[16:44] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) Quit (Read error: Connection reset by peer)
[16:45] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) has joined #ceph
[16:46] * Sysadmin88 (~IceChat77@94.12.240.104) has joined #ceph
[16:47] * cok (~chk@2a02:2350:18:1010:ddb3:f017:e8ff:97e9) has joined #ceph
[16:47] * fghaas (~florian@185.15.236.4) has joined #ceph
[16:52] * cok (~chk@2a02:2350:18:1010:ddb3:f017:e8ff:97e9) Quit ()
[16:52] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) Quit (Read error: Connection reset by peer)
[16:53] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) has joined #ceph
[16:54] * rljohnsn (~rljohnsn@ns25.8x8.com) has joined #ceph
[16:54] * dmsimard is now known as dmsimard_away
[16:58] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) Quit (Read error: Connection reset by peer)
[16:59] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) has joined #ceph
[16:59] * bandrus (~brian@230.sub-70-211-80.myvzw.com) has joined #ceph
[17:00] * rljohnsn (~rljohnsn@ns25.8x8.com) Quit (Quit: Leaving.)
[17:02] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) Quit (Read error: Connection reset by peer)
[17:03] * analbeard (~shw@support.memset.com) Quit (Quit: Leaving.)
[17:03] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) has joined #ceph
[17:04] * tupper_ (~tcole@108-83-203-37.lightspeed.rlghnc.sbcglobal.net) Quit (Ping timeout: 480 seconds)
[17:04] * rljohnsn (~rljohnsn@ns25.8x8.com) has joined #ceph
[17:05] * bitserker (~toni@213.229.187.103) has joined #ceph
[17:07] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) Quit (Read error: Connection reset by peer)
[17:07] * i_m (~ivan.miro@deibp9eh1--blueice1n2.emea.ibm.com) Quit (Quit: Leaving.)
[17:07] * xarses (~andreww@12.164.168.117) has joined #ceph
[17:07] * i_m (~ivan.miro@deibp9eh1--blueice1n2.emea.ibm.com) has joined #ceph
[17:08] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) has joined #ceph
[17:10] * amote (~amote@121.244.87.116) Quit (Quit: Leaving)
[17:11] * reed (~reed@75-101-54-131.dsl.static.fusionbroadband.com) has joined #ceph
[17:11] * linjan (~linjan@176.195.18.148) Quit (Ping timeout: 480 seconds)
[17:12] * shaunm (~shaunm@nat-pool-brq-t.redhat.com) Quit (Ping timeout: 482 seconds)
[17:13] * tupper_ (~tcole@rtp-isp-nat1.cisco.com) has joined #ceph
[17:13] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) Quit (Read error: Connection reset by peer)
[17:15] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) has joined #ceph
[17:15] * i_m (~ivan.miro@deibp9eh1--blueice1n2.emea.ibm.com) Quit (Ping timeout: 480 seconds)
[17:17] * puffy (~puffy@50.185.218.255) has joined #ceph
[17:20] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) Quit (Read error: Connection reset by peer)
[17:20] <fghaas> hey everyone, random question I've been meaning to ask for some time: what is the expected behavior in case your crushmap has the same bucket reference in multiple places?
[17:21] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) has joined #ceph
[17:21] <VisBits> fghaas https://www.youtube.com/watch?v=kEpLncBG_Nw
[17:22] <fghaas> say for example you have a host, "foo", and two different crush rulesets, and you're referring to foo from, say, rack=23 and aisle=left, where one ruleset is based on racks and one on aisles (for a contrived example)
[17:23] <fghaas> is that considered okay?
[17:24] <fghaas> iow, is it just a ruleset that is considered strictly monohierarchical, or the whole crushmap?
[17:24] * shaunm (~shaunm@nat-pool-brq-u.redhat.com) has joined #ceph
[17:24] <Be-El> fghaas: as long as the same crush ruleset does not allow the same entry to appear more than once, it should be ok
[17:25] <fghaas> define "should be", please?
[17:25] * sudocat1 (~davidi@192.185.1.20) has joined #ceph
[17:25] <Be-El> fghaas: "should be" like in "i'm not a ceph developer"
[17:25] * vakulkar (~vakulkar@c-50-185-132-102.hsd1.ca.comcast.net) has joined #ceph
[17:26] <fghaas> Be-El: well, assumptions I have no shortage of, myself :)
[17:27] <Be-El> the problem you might run into with overlapping rulesets is the uneven distribution of data amoung osds
[17:31] * j^2 (sid14252@id-14252.brockwell.irccloud.com) has joined #ceph
[17:31] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) Quit (Read error: Connection reset by peer)
[17:31] <j^2> Hey everyone! it semes that the ceph-cookbook is at 0.2.0, and the gitub app is at 0.2.1 any chance it???s gonna get push to supermarket soon?
[17:32] <alfredodeza> supermarket?
[17:32] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) has joined #ceph
[17:32] * rljohnsn (~rljohnsn@ns25.8x8.com) Quit (Quit: Leaving.)
[17:32] <j^2> http://supermarket.chef.io
[17:34] <j^2> yeah i???m representing the openstack-chef project, and we???re downstream and would love to have it :)
[17:34] * sudocat1 (~davidi@192.185.1.20) Quit (Quit: Leaving.)
[17:34] * sudocat (~davidi@192.185.1.20) has joined #ceph
[17:34] * puffy (~puffy@50.185.218.255) Quit (Quit: Leaving.)
[17:35] * togdon (~togdon@74.121.28.6) has joined #ceph
[17:36] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) Quit (Read error: Connection reset by peer)
[17:38] * sputnik13 (~sputnik13@c-73-193-97-20.hsd1.wa.comcast.net) Quit (Quit: My MacBook has gone to sleep. ZZZzzz???)
[17:38] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) has joined #ceph
[17:40] * TMM (~hp@sams-office-nat.tomtomgroup.com) Quit (Quit: Ex-Chat)
[17:40] * sputnik13 (~sputnik13@c-73-193-97-20.hsd1.wa.comcast.net) has joined #ceph
[17:41] * alram (~alram@LAubervilliers-656-1-17-4.w217-128.abo.wanadoo.fr) Quit (Ping timeout: 480 seconds)
[17:42] * sputnik13 (~sputnik13@c-73-193-97-20.hsd1.wa.comcast.net) Quit ()
[17:43] * jclm1 (~jclm@ip24-253-45-236.lv.lv.cox.net) has joined #ceph
[17:44] * ngoswami is now known as ngoswami|mtng
[17:46] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) Quit (Read error: Connection reset by peer)
[17:49] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) has joined #ceph
[17:49] * jclm (~jclm@ip24-253-45-236.lv.lv.cox.net) Quit (Ping timeout: 480 seconds)
[17:54] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) Quit (Read error: Connection reset by peer)
[17:55] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) has joined #ceph
[17:56] * alram (~alram@ppp-seco11pa2-46-193-132-162.wb.wifirst.net) has joined #ceph
[17:57] * primechuck (~primechuc@host-95-2-129.infobunker.com) has joined #ceph
[17:58] * rmoe (~quassel@173-228-89-134.dsl.static.fusionbroadband.com) Quit (Ping timeout: 480 seconds)
[18:00] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) Quit (Read error: Connection reset by peer)
[18:01] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) has joined #ceph
[18:05] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) Quit (Read error: Connection reset by peer)
[18:06] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) has joined #ceph
[18:08] * rljohnsn (~rljohnsn@ns25.8x8.com) has joined #ceph
[18:09] * rohanm (~rohanm@c-67-168-194-197.hsd1.or.comcast.net) Quit (Ping timeout: 480 seconds)
[18:09] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) Quit (Read error: Connection reset by peer)
[18:10] * rmoe (~quassel@12.164.168.117) has joined #ceph
[18:11] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) has joined #ceph
[18:13] * puffy (~puffy@50.185.218.255) has joined #ceph
[18:13] * georgem (~Adium@fwnat.oicr.on.ca) has joined #ceph
[18:13] <avozza> hi, is this a good place to ask about calamari?
[18:13] * rljohnsn (~rljohnsn@ns25.8x8.com) Quit (Quit: Leaving.)
[18:13] <VisBits> better chance of getting a reply asking a question than if you can ask a question
[18:14] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[18:14] <avozza> you got a point :)
[18:14] * puffy (~puffy@50.185.218.255) Quit ()
[18:14] <avozza> just wondering if I could install calamari on an existing postgres database
[18:14] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[18:15] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[18:15] <avozza> calamari-ctl initialize doesn't let you specify much
[18:15] * sleinen1 (~Adium@2001:620:0:82::10a) has joined #ceph
[18:17] * rljohnsn (~rljohnsn@ns25.8x8.com) has joined #ceph
[18:19] * dmsimard_away is now known as dmsimard
[18:19] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) Quit (Read error: Connection reset by peer)
[18:21] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) has joined #ceph
[18:22] * sleinen (~Adium@2001:620:0:2d:7ed1:c3ff:fedc:3223) Quit (Ping timeout: 480 seconds)
[18:24] * Lyncos (~lyncos@208.71.184.41) has joined #ceph
[18:24] <saltlake> test
[18:25] <Lyncos> Hi everyone.. I got a quick question... I did follow the whole osd removal procedure. but when I deleted my empty crush bucket (the host) my cluster started to re balance... any one know why ?
[18:25] <pmatulis> test successful saltlake
[18:25] <saltlake> pmaulis: thanks for some reason could not send messages to another channel.. pls ignore thank you
[18:27] <Be-El> Lyncos: what was the state of the bucket prior to removal? was there still a weight associated to it?
[18:27] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) Quit (Read error: Connection reset by peer)
[18:27] <Lyncos> Be-El I guess yes
[18:27] <Lyncos> I should have put a weight of 0 before ? even if there is no more OSD ?
[18:27] <Lyncos> hmm
[18:27] <Be-El> Lyncos: there should have been no weight for that bucket
[18:28] <Lyncos> the weight was 0 sorry I double checked
[18:28] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) has joined #ceph
[18:28] <Lyncos> I removed all the osd from the pool first .. then removed the bucket.
[18:28] <Lyncos> health HEALTH_WARN 2569 pgs backfill; 77 pgs backfilling; 1 pgs peering; 2648 pgs stuck unclean; recovery -2/43650203 objects degraded (-0.000%); 5728654/43650203 objects misplaced (13.124%)
[18:28] <Lyncos> now I get that status
[18:29] <Be-El> Lyncos: and there was no rebalancing after the osd were removed? did you set their weight to 0 prior to removal?
[18:29] <Lyncos> I did osd out for all of them and let the cluster rebalance
[18:29] <Lyncos> Is it the same as setting the weight to 0 ?
[18:30] * alram (~alram@ppp-seco11pa2-46-193-132-162.wb.wifirst.net) Quit (Quit: leaving)
[18:30] <Be-El> nope. as far as i known, setting the osd to out does not influence their weight in the crush map. it just redistributed the data since the osds are not available anymore
[18:30] <Lyncos> So I should set them out first then weight them to 0 (not reweight)
[18:30] <Be-El> Lyncos: upon removal, the weight of the host buckets (and other buckets upstream) changes, influencing the whole cluster
[18:31] <Be-El> Lyncos: i usually set their weight to 0, let the cluster rebalance, and remove the osds afterwards
[18:31] <Lyncos> ok no need to set them out
[18:31] <Lyncos> just set their weight to 0
[18:32] <Be-El> Lyncos: no, it's an additional step
[18:32] <Lyncos> can you tell me what command you are using ?
[18:32] <Be-El> Lyncos: reweight, wait, out, kill, remove
[18:32] * sleinen1 (~Adium@2001:620:0:82::10a) Quit (Ping timeout: 480 seconds)
[18:32] <Lyncos> but.. reweight isn't the same as osd out ? I tought osd out reweight them to 0
[18:33] <Lyncos> maybe I missed something
[18:33] <Be-El> Lyncos: the command is 'ceph osd crush reweight <osd> <weight>'
[18:33] <Lyncos> ok will try that
[18:34] <Lyncos> I'll let re-balance and I have another node to rebuild .. will test it
[18:34] <Be-El> Lyncos: setting an osd to out only changes the state of the osd. it does not change the crush tree
[18:34] <Lyncos> Ok thanks I'll try that I guess that was the step I missed up...
[18:34] * thomnico (~thomnico@88.131.14.170) has joined #ceph
[18:35] <Be-El> Lyncos: that's why the default way results in two cluster rebalances: first on osd out (osd is not available, data has to be redistributed), second on remove (weights in tree have changed)
[18:35] <Lyncos> ok .. so I'm better to reweight them before putting them out... so only 1 re-balance occurs
[18:35] <Lyncos> I would like to do it in a more controlled way
[18:35] <Be-El> exactly
[18:36] * thomnico (~thomnico@88.131.14.170) Quit ()
[18:36] * shaunm (~shaunm@nat-pool-brq-u.redhat.com) Quit (Ping timeout: 480 seconds)
[18:36] <Be-El> replacing a failed drive is different story
[18:39] <Lyncos> ah ok this in a decommissioning scenario
[18:39] <Be-El> do you have several osds in the affected hosts?
[18:39] <Lyncos> replacing failed drive.. just doing out would be fine ?
[18:40] <Lyncos> yeah I'm decomm that host it has 24 drives...
[18:40] * thomnico (~thomnico@88.131.14.170) has joined #ceph
[18:40] * rljohnsn (~rljohnsn@ns25.8x8.com) Quit (Quit: Leaving.)
[18:40] <Lyncos> at same time I'm writing our internal precedure doc
[18:41] <VisBits> Lyncos automate that shit
[18:41] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) Quit (Read error: Connection reset by peer)
[18:41] <Be-El> don't remind me of docs, i'll have to write them in the near future...
[18:41] <Lyncos> VisBits that is why doc is here.. I'll get someone automate it for me :-)
[18:41] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) has joined #ceph
[18:41] <fghaas> hmmm... if you checked the OSD filestore in 3 different OSDs, and the primary and one replica were empty, and the second replica had data, and the PG were marked degraded, is there any reasonable explanation other than that the objects have simply been deleted and the second replica is waiting to catch up on that deletion?
[18:42] <fghaas> s/OSD filestore/OSD filestores for a PG in a pool of size 3,/
[18:42] <kraken> fghaas meant to say: hmmm... if you checked the OSD filestores for a PG in a pool of size 3, in 3 different OSDs, and the primary and one replica were empty, and the second replica had data, and the PG were marked degraded, is there any reasonable explanation other than that the objects have simply been deleted and the second replica is waiting to catch up on
[18:42] <kraken> that deletion?
[18:43] * oro_ (~oro@2001:620:20:16:9858:9d3:b3e9:fc05) Quit (Ping timeout: 480 seconds)
[18:45] * oro (~oro@2001:620:20:16:9858:9d3:b3e9:fc05) Quit (Ping timeout: 480 seconds)
[18:45] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) Quit (Read error: Connection reset by peer)
[18:45] * georgem (~Adium@fwnat.oicr.on.ca) Quit (Quit: Leaving.)
[18:46] * ngoswami|mtng (~ngoswami@121.244.87.116) Quit (Quit: Leaving)
[18:47] * mykola (~Mikolaj@91.225.201.255) has joined #ceph
[18:49] * Concubidated (~Adium@2607:f298:b:635:68b4:7a8:5742:d6ec) has joined #ceph
[18:49] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) has joined #ceph
[18:49] * Kupo1 (~tyler.wil@23.111.254.159) Quit (Read error: Connection reset by peer)
[18:49] * togdon (~togdon@74.121.28.6) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[18:50] * rotbeard (~redbeard@aftr-37-201-193-222.unity-media.net) has joined #ceph
[18:54] * marrusl (~mark@cpe-24-90-46-248.nyc.res.rr.com) Quit (Quit: sync && halt)
[18:55] * bandrus1 (~brian@230.sub-70-211-80.myvzw.com) has joined #ceph
[18:55] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) Quit (Read error: Connection reset by peer)
[18:55] * vilobhmm (~vilobhmm@nat-dip33-wl-g.cfw-a-gci.corp.yahoo.com) has joined #ceph
[18:55] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) has joined #ceph
[18:55] * togdon (~togdon@74.121.28.6) has joined #ceph
[18:57] * rljohnsn (~rljohnsn@ns25.8x8.com) has joined #ceph
[18:57] <fghaas> scuttlemonkey: any chance you could resurrect http://lists.ceph.com/pipermail/ceph-users-ceph.com/ ?
[18:58] * cholcombe973 (~chris@7208-76ef-ff1f-ed2f-329a-f002-3420-2062.6rd.ip6.sonic.net) has joined #ceph
[19:00] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) Quit (Read error: Connection reset by peer)
[19:00] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) has joined #ceph
[19:01] * bandrus (~brian@230.sub-70-211-80.myvzw.com) Quit (Ping timeout: 480 seconds)
[19:01] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) Quit (Read error: Connection reset by peer)
[19:03] * Kupo1 (~tyler.wil@23.111.254.159) has joined #ceph
[19:03] * puffy (~puffy@25.sub-174-240-5.myvzw.com) has joined #ceph
[19:03] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) has joined #ceph
[19:05] * angdraug (~angdraug@12.164.168.117) has joined #ceph
[19:05] * thomnico (~thomnico@88.131.14.170) Quit (Remote host closed the connection)
[19:07] * togdon (~togdon@74.121.28.6) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[19:11] * archiestengol (~chatzilla@c-50-183-112-236.hsd1.co.comcast.net) Quit (Read error: Connection reset by peer)
[19:14] * qhartman (~qhartman@den.direwolfdigital.com) Quit (Quit: Ex-Chat)
[19:15] * guerby (~guerby@ip165-ipv6.tetaneutral.net) Quit (Quit: Leaving)
[19:15] * guerby (~guerby@ip165-ipv6.tetaneutral.net) has joined #ceph
[19:15] * togdon (~togdon@74.121.28.6) has joined #ceph
[19:17] * bandrus1 (~brian@230.sub-70-211-80.myvzw.com) Quit (Ping timeout: 480 seconds)
[19:17] * lalatenduM (~lalatendu@122.171.99.96) has joined #ceph
[19:18] * vbellur (~vijay@122.167.79.204) Quit (Ping timeout: 480 seconds)
[19:19] * kawa2014 (~kawa@89.184.114.246) Quit (Quit: Leaving)
[19:19] * georgem (~Adium@fwnat.oicr.on.ca) has joined #ceph
[19:20] * rljohnsn (~rljohnsn@ns25.8x8.com) Quit (Quit: Leaving.)
[19:21] * fghaas (~florian@185.15.236.4) Quit (Quit: Leaving.)
[19:22] * fghaas (~florian@185.15.236.4) has joined #ceph
[19:22] * togdon (~togdon@74.121.28.6) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[19:22] * fghaas (~florian@185.15.236.4) Quit ()
[19:26] * georgem (~Adium@fwnat.oicr.on.ca) Quit (Quit: Leaving.)
[19:27] * bandrus (~brian@184.sub-70-211-83.myvzw.com) has joined #ceph
[19:34] * rljohnsn (~rljohnsn@ns25.8x8.com) has joined #ceph
[19:35] * georgem (~Adium@fwnat.oicr.on.ca) has joined #ceph
[19:35] * jordanP (~jordan@213.215.2.194) Quit (Remote host closed the connection)
[19:35] * bitserker (~toni@213.229.187.103) Quit (Ping timeout: 480 seconds)
[19:39] <cholcombe973> ceph: question about crush. If I'm understanding this correctly I can just set the location information in the ceph.conf file and when the daemon starts it'll put itself into the right place in the crush map. Is that correct?
[19:44] * sputnik13 (~sputnik13@74.202.214.170) has joined #ceph
[19:47] * nils_ (~nils@doomstreet.collins.kg) Quit (Quit: This computer has gone to sleep)
[19:47] * LeaChim (~LeaChim@host86-159-114-39.range86-159.btcentralplus.com) has joined #ceph
[19:50] * lalatenduM (~lalatendu@122.171.99.96) Quit (Ping timeout: 480 seconds)
[19:50] * Nacer (~Nacer@252-87-190-213.intermediasud.com) Quit (Read error: Connection reset by peer)
[19:50] * Nacer (~Nacer@252-87-190-213.intermediasud.com) has joined #ceph
[19:51] * Nacer_ (~Nacer@252-87-190-213.intermediasud.com) has joined #ceph
[19:52] * smokedmeets (~smokedmee@c-67-174-241-112.hsd1.ca.comcast.net) has left #ceph
[19:53] * partygoblin (~smokedmee@c-67-174-241-112.hsd1.ca.comcast.net) has joined #ceph
[19:54] * alram (~alram@ppp-seco11pa2-46-193-132-162.wb.wifirst.net) has joined #ceph
[19:56] * oro_ (~oro@80-219-254-208.dclient.hispeed.ch) has joined #ceph
[19:58] * Nacer (~Nacer@252-87-190-213.intermediasud.com) Quit (Ping timeout: 480 seconds)
[19:59] * t0rn (~ssullivan@2607:fad0:32:a02:d227:88ff:fe02:9896) Quit (Quit: Leaving.)
[19:59] * Nacer_ (~Nacer@252-87-190-213.intermediasud.com) Quit (Ping timeout: 480 seconds)
[19:59] * oro (~oro@80-219-254-208.dclient.hispeed.ch) has joined #ceph
[20:00] * bandrus1 (~brian@184.sub-70-211-83.myvzw.com) has joined #ceph
[20:00] <scuttlemonkey> fghaas: strange about the user list
[20:04] * sputnik13 (~sputnik13@74.202.214.170) Quit (Quit: My MacBook has gone to sleep. ZZZzzz???)
[20:04] * sputnik13 (~sputnik13@74.202.214.170) has joined #ceph
[20:07] * bandrus (~brian@184.sub-70-211-83.myvzw.com) Quit (Ping timeout: 480 seconds)
[20:07] * davidz (~davidz@2605:e000:1313:8003:15ad:82bc:ee4f:93d1) Quit (Ping timeout: 480 seconds)
[20:09] * davidz (~davidz@2605:e000:1313:a104:1b0:d48e:a9ab:fb51) has joined #ceph
[20:11] * fghaas (~florian@185.15.236.4) has joined #ceph
[20:12] * alram (~alram@ppp-seco11pa2-46-193-132-162.wb.wifirst.net) Quit (Quit: leaving)
[20:19] * davidz (~davidz@2605:e000:1313:a104:1b0:d48e:a9ab:fb51) Quit (Ping timeout: 480 seconds)
[20:21] * davidz (~davidz@2605:e000:1313:8003:1dc0:2023:672e:59e6) has joined #ceph
[20:22] * yogh (~yogh@sol.kvlt.net) has joined #ceph
[20:22] * danieagle (~Daniel@201-95-103-54.dsl.telesp.net.br) Quit (Quit: Obrigado por Tudo! :-) inte+ :-))
[20:23] <primechuck> What would the possible issues be with having a monitor 40 - 100ms away from the OSDs?
[20:24] * oro_ (~oro@80-219-254-208.dclient.hispeed.ch) Quit (Ping timeout: 480 seconds)
[20:24] * vata (~vata@208.88.110.46) has joined #ceph
[20:25] <scuttlemonkey> fghaas: dreamhost admins looking at lists
[20:25] <scuttlemonkey> there is a hardware migration going on though, so that may be part of it
[20:25] <scuttlemonkey> I suspect it's just a permissions problem that should be resolved
[20:25] <scuttlemonkey> will know more shortly
[20:25] <fghaas> scuttlemonkey: great, thanks!
[20:25] * oro (~oro@80-219-254-208.dclient.hispeed.ch) Quit (Ping timeout: 480 seconds)
[20:27] * L2SHO (~L2SHO@office-nat.choopa.net) has joined #ceph
[20:29] * davidz1 (~davidz@2605:e000:1313:8003:7d8e:62dc:8733:6fc0) has joined #ceph
[20:30] * lcurtis (~lcurtis@47.19.105.250) has joined #ceph
[20:32] <fghaas> getting back to my earlier question. recovery has now progressed past the affected PG and I am seeing something that I can't make head nor tail of. So the PG was mapped to OSDs [A,B,C], where during the recovery A and B were both empty, and C had data. A was (and still is) the primary .
[20:32] * davidz2 (~davidz@cpe-23-242-189-171.socal.res.rr.com) has joined #ceph
[20:32] <fghaas> after the recovery, data is identical on A, B, C. Now what does that mean *during* the recovery?
[20:33] <fghaas> if A was primary, clients would hit A. A, however, had no data. Would it pretend that the objects did not exist?
[20:34] * davidz (~davidz@2605:e000:1313:8003:1dc0:2023:672e:59e6) Quit (Ping timeout: 480 seconds)
[20:37] * danieljh (~daniel@0001b4e9.user.oftc.net) has joined #ceph
[20:39] * swizgard (~swizgard@gate.gxp-brain.fta-berlin.de) Quit (Quit: leaving)
[20:39] * derjohn_mob (~aj@94.119.1.11) Quit (Ping timeout: 480 seconds)
[20:39] * houkouonchi-work (~linux@2607:f298:b:635:225:90ff:fe39:38ce) has joined #ceph
[20:40] * davidz1 (~davidz@2605:e000:1313:8003:7d8e:62dc:8733:6fc0) Quit (Ping timeout: 480 seconds)
[20:40] * davidz2 (~davidz@cpe-23-242-189-171.socal.res.rr.com) Quit (Quit: Leaving.)
[20:42] * haomaiwang (~haomaiwan@115.218.155.68) Quit (Ping timeout: 480 seconds)
[20:42] * oro (~oro@80-219-254-208.dclient.hispeed.ch) has joined #ceph
[20:43] * primechuck (~primechuc@host-95-2-129.infobunker.com) Quit (Quit: Leaving...)
[20:44] * haomaiwang (~haomaiwan@115.218.155.68) has joined #ceph
[20:45] * oro_ (~oro@80-219-254-208.dclient.hispeed.ch) has joined #ceph
[20:47] <fghaas> ok, let's try the next question. :) assuming osd max backfills = 1, and you had about 20 PGs about to backfill in your cluster, and those PGs had practically no overlap in acting OSDs, why would you only see a single backfilling operation in the entire cluster, with all other PGs waiting to backfill?
[20:48] * rljohnsn (~rljohnsn@ns25.8x8.com) Quit (Quit: Leaving.)
[20:48] * davidzlap (~Adium@2605:e000:1313:8003:ade7:a884:a244:6462) has joined #ceph
[20:48] * Nacer (~Nacer@2001:41d0:fe82:7200:5d02:6f2:3740:2bb) has joined #ceph
[20:49] * DavidThunder (~Thunderbi@2605:e000:1313:8003:ade7:a884:a244:6462) has joined #ceph
[20:49] * Nacer (~Nacer@2001:41d0:fe82:7200:5d02:6f2:3740:2bb) Quit (Remote host closed the connection)
[20:50] * togdon (~togdon@74.121.28.6) has joined #ceph
[20:52] * avozza_ (~avozza@83.162.204.36) has joined #ceph
[20:52] * avozza (~avozza@83.162.204.36) Quit (Read error: Connection reset by peer)
[20:54] * togdon_ (~togdon@74.121.28.6) has joined #ceph
[20:54] * togdon (~togdon@74.121.28.6) Quit (Read error: Connection reset by peer)
[20:57] * primechuck (~primechuc@host-95-2-129.infobunker.com) has joined #ceph
[20:57] * linjan (~linjan@80.178.220.195.adsl.012.net.il) has joined #ceph
[21:01] * DavidThunder (~Thunderbi@2605:e000:1313:8003:ade7:a884:a244:6462) Quit (Quit: DavidThunder)
[21:01] * DavidThunder (~Thunderbi@2605:e000:1313:8003:ade7:a884:a244:6462) has joined #ceph
[21:07] * georgem (~Adium@fwnat.oicr.on.ca) Quit (Quit: Leaving.)
[21:07] * togdon_ (~togdon@74.121.28.6) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[21:08] * davidzlap (~Adium@2605:e000:1313:8003:ade7:a884:a244:6462) Quit (Ping timeout: 480 seconds)
[21:08] * georgem (~Adium@fwnat.oicr.on.ca) has joined #ceph
[21:08] * DavidThunder1 (~Thunderbi@2605:e000:1313:8003:ade7:a884:a244:6462) has joined #ceph
[21:09] * DavidThunder (~Thunderbi@2605:e000:1313:8003:ade7:a884:a244:6462) Quit (Ping timeout: 480 seconds)
[21:10] * davidzlap (~Adium@cpe-23-242-189-171.socal.res.rr.com) has joined #ceph
[21:11] * DavidThunder (~Thunderbi@cpe-23-242-189-171.socal.res.rr.com) has joined #ceph
[21:13] * georgem (~Adium@fwnat.oicr.on.ca) Quit ()
[21:14] * georgem (~Adium@fwnat.oicr.on.ca) has joined #ceph
[21:15] * Be-El (~quassel@fb08-bcf-pc01.computational.bio.uni-giessen.de) Quit (Remote host closed the connection)
[21:15] * fcpesto (~fcpesto@129.82.10.18) has joined #ceph
[21:16] * togdon (~togdon@74.121.28.6) has joined #ceph
[21:16] * DavidThunder (~Thunderbi@cpe-23-242-189-171.socal.res.rr.com) Quit (Quit: DavidThunder)
[21:16] * DavidThunder1 (~Thunderbi@2605:e000:1313:8003:ade7:a884:a244:6462) Quit (Ping timeout: 480 seconds)
[21:19] * vasu_desk (~vasu@c-50-185-132-102.hsd1.ca.comcast.net) has joined #ceph
[21:22] * rljohnsn (~rljohnsn@ns25.8x8.com) has joined #ceph
[21:23] * linjan (~linjan@80.178.220.195.adsl.012.net.il) Quit (Ping timeout: 480 seconds)
[21:25] * fcpesto (~fcpesto@129.82.10.18) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[21:26] * togdon (~togdon@74.121.28.6) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[21:33] * linjan (~linjan@80.178.220.195.adsl.012.net.il) has joined #ceph
[21:33] * L2SHO (~L2SHO@office-nat.choopa.net) Quit (Quit: Leaving)
[21:38] * togdon (~togdon@74.121.28.6) has joined #ceph
[21:40] * L2SHO (~L2SHO@2001:19f0:1000:5123:8c84:23f:8ca:f675) has joined #ceph
[21:41] * linjan (~linjan@80.178.220.195.adsl.012.net.il) Quit (Ping timeout: 480 seconds)
[21:48] * partygoblin (~smokedmee@c-67-174-241-112.hsd1.ca.comcast.net) Quit (Quit: partygoblin)
[21:49] * rljohnsn (~rljohnsn@ns25.8x8.com) Quit (Quit: Leaving.)
[21:51] * togdon (~togdon@74.121.28.6) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[21:51] * fcpesto (~fcpesto@129.82.10.18) has joined #ceph
[21:52] * fcpesto (~fcpesto@129.82.10.18) Quit ()
[21:55] * puffy (~puffy@25.sub-174-240-5.myvzw.com) Quit (Ping timeout: 480 seconds)
[21:55] * oro (~oro@80-219-254-208.dclient.hispeed.ch) Quit (Ping timeout: 480 seconds)
[21:55] * oro_ (~oro@80-219-254-208.dclient.hispeed.ch) Quit (Ping timeout: 480 seconds)
[21:59] * rljohnsn (~rljohnsn@ns25.8x8.com) has joined #ceph
[22:01] * togdon (~togdon@74.121.28.6) has joined #ceph
[22:02] * togdon (~togdon@74.121.28.6) Quit ()
[22:03] * georgem (~Adium@fwnat.oicr.on.ca) has left #ceph
[22:10] * togdon (~togdon@74.121.28.6) has joined #ceph
[22:14] * smokedmeets (~smokedmee@c-67-174-241-112.hsd1.ca.comcast.net) has joined #ceph
[22:25] * cookednoodles (~eoin@89-93-153-201.hfc.dyn.abo.bbox.fr) has joined #ceph
[22:25] <flaf> fghaas: sorry, no answer. Are you sure that A was primary during the recovery? In this case, me too, I don't understand.
[22:27] * togdon (~togdon@74.121.28.6) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[22:30] <fghaas> flaf, yes it definitely was; I checked with ceph pg query while the PG was queued for recovery
[22:30] * badone (~brad@CPE-121-215-241-179.static.qld.bigpond.net.au) has joined #ceph
[22:32] <flaf> And how have you checked that A was empty?
[22:34] * saltlake (~saltlake@12.250.199.170) Quit (Quit: Nettalk6 - www.ntalk.de)
[22:34] * rotbeard (~redbeard@aftr-37-201-193-222.unity-media.net) Quit (Quit: Leaving)
[22:37] * rljohnsn (~rljohnsn@ns25.8x8.com) Quit (Quit: Leaving.)
[22:37] * jrankin (~jrankin@d53-64-170-236.nap.wideopenwest.com) Quit (Quit: Leaving)
[22:38] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit (Ping timeout: 480 seconds)
[22:39] * rljohnsn (~rljohnsn@ns25.8x8.com) has joined #ceph
[22:42] * smokedmeets (~smokedmee@c-67-174-241-112.hsd1.ca.comcast.net) Quit (Quit: smokedmeets)
[22:42] * saltlake (~saltlake@12.250.199.170) has joined #ceph
[22:42] * _br_ (~bjoern_of@213-239-215-232.clients.your-server.de) Quit (Ping timeout: 480 seconds)
[22:49] * badone (~brad@CPE-121-215-241-179.static.qld.bigpond.net.au) Quit (Ping timeout: 480 seconds)
[22:49] * _br_ (~bjoern_of@213-239-215-232.clients.your-server.de) has joined #ceph
[22:49] * sjm (~sjm@pool-98-109-11-113.nwrknj.fios.verizon.net) has left #ceph
[22:52] * rljohnsn (~rljohnsn@ns25.8x8.com) Quit (Quit: Leaving.)
[22:53] * mykola (~Mikolaj@91.225.201.255) Quit (Quit: away)
[22:54] * dyasny (~dyasny@173.231.115.58) Quit (Ping timeout: 480 seconds)
[22:57] * rotbeard (~redbeard@2a02:908:df10:d300:76f0:6dff:fe3b:994d) has joined #ceph
[23:01] * rljohnsn (~rljohnsn@ns25.8x8.com) has joined #ceph
[23:04] * L2SHO (~L2SHO@2001:19f0:1000:5123:8c84:23f:8ca:f675) Quit (Quit: Leaving)
[23:05] * L2SHO (~L2SHO@2001:19f0:1000:5123:8c84:23f:8ca:f675) has joined #ceph
[23:06] <yogh> hi #ceph, we are having some issues with our radosgw setup - v0.87. We created user "miketest" and bucket "mixtapes" and put some objects in it. Everything was ok until recently, we are getting 403/AccessDenied when trying to put additional objects. Does anyone have suggestions on how to troubleshoot and/or resolve the issue? Do I have bad user/bucket permissions or something like that? I pasted radosgw client log on <http://hastebin.com/udezitiyek.xml>
[23:07] * puffy (~puffy@235.sub-174-240-8.myvzw.com) has joined #ceph
[23:08] * saltlake (~saltlake@12.250.199.170) Quit (Ping timeout: 480 seconds)
[23:08] * hasues (~hazuez@108-236-232-243.lightspeed.knvltn.sbcglobal.net) has joined #ceph
[23:09] * hasues (~hazuez@108-236-232-243.lightspeed.knvltn.sbcglobal.net) has left #ceph
[23:09] * smokedmeets (~smokedmee@c-67-174-241-112.hsd1.ca.comcast.net) has joined #ceph
[23:10] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[23:10] * _br_ (~bjoern_of@213-239-215-232.clients.your-server.de) Quit (Ping timeout: 480 seconds)
[23:11] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[23:11] * vilobhmm (~vilobhmm@nat-dip33-wl-g.cfw-a-gci.corp.yahoo.com) Quit (Quit: Away)
[23:12] <VisBits> how does rbdmap work if you have multiple clusters..
[23:13] * vilobhmm (~vilobhmm@nat-dip33-wl-g.cfw-a-gci.corp.yahoo.com) has joined #ceph
[23:14] * sleinen (~Adium@84-72-160-233.dclient.hispeed.ch) has joined #ceph
[23:15] * badone (~brad@CPE-121-215-241-179.static.qld.bigpond.net.au) has joined #ceph
[23:15] * puffy (~puffy@235.sub-174-240-8.myvzw.com) Quit (Ping timeout: 480 seconds)
[23:16] * TMM (~hp@178-84-46-106.dynamic.upc.nl) has joined #ceph
[23:17] * sleinen1 (~Adium@2001:620:0:82::101) has joined #ceph
[23:17] * Kupo1 (~tyler.wil@23.111.254.159) Quit (Read error: Connection reset by peer)
[23:20] * Hell_Fire_ (~hellfire@123-243-155-184.static.tpgi.com.au) has joined #ceph
[23:21] * Kupo1 (~tyler.wil@23.111.254.159) has joined #ceph
[23:21] * Hell_Fire (~hellfire@123-243-155-184.static.tpgi.com.au) Quit (Read error: Connection reset by peer)
[23:23] * sleinen (~Adium@84-72-160-233.dclient.hispeed.ch) Quit (Ping timeout: 480 seconds)
[23:25] * fghaas (~florian@185.15.236.4) Quit (Quit: Leaving.)
[23:25] * smokedmeets (~smokedmee@c-67-174-241-112.hsd1.ca.comcast.net) Quit (Quit: smokedmeets)
[23:29] * brad_mssw (~brad@66.129.88.50) Quit (Quit: Leaving)
[23:29] * cookednoodles (~eoin@89-93-153-201.hfc.dyn.abo.bbox.fr) Quit (Quit: Ex-Chat)
[23:34] * thb (~me@0001bd58.user.oftc.net) Quit (Quit: Leaving.)
[23:34] <yogh> maybe AccessControlPolicy/Permission contains read but does not contain write? how can we update that information?
[23:35] * _br_ (~bjoern_of@213-239-215-232.clients.your-server.de) has joined #ceph
[23:35] * togdon (~togdon@74.121.28.6) has joined #ceph
[23:37] * badone (~brad@CPE-121-215-241-179.static.qld.bigpond.net.au) Quit (Quit: Kirk out)
[23:37] * rotbart (~redbeard@2a02:908:df10:d300:6267:20ff:feb7:c20) has joined #ceph
[23:39] <flaf> yogh: maybe try "radosgw-admin bucket stats --bucket=$bucket_name". You will have the owner of the bucket.
[23:40] <yogh> Cool. I think Policy read should be policy full_control... checking to see how we can update it
[23:44] * Rickus_ (~Rickus@office.protected.ca) Quit (Read error: Connection reset by peer)
[23:47] * rotbart (~redbeard@2a02:908:df10:d300:6267:20ff:feb7:c20) Quit (Quit: Leaving)
[23:48] * Rickus (~Rickus@office.protected.ca) has joined #ceph
[23:48] * badone (~brad@CPE-121-215-241-179.static.qld.bigpond.net.au) has joined #ceph
[23:51] * joshd (~joshd@sccc-66-78-236-243.smartcity.com) has joined #ceph
[23:52] * sleinen1 (~Adium@2001:620:0:82::101) Quit (Ping timeout: 480 seconds)
[23:52] * _br_ (~bjoern_of@213-239-215-232.clients.your-server.de) Quit (Ping timeout: 480 seconds)
[23:53] * rljohnsn (~rljohnsn@ns25.8x8.com) Quit (Quit: Leaving.)
[23:56] * rljohnsn (~rljohnsn@ns25.8x8.com) has joined #ceph
[23:59] * segutier (~segutier@c-24-6-218-139.hsd1.ca.comcast.net) has joined #ceph

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.