#ceph IRC Log

Index

IRC Log for 2016-07-18

Timestamps are in GMT/BST.

[0:00] * theTrav (~theTrav@ipc032.ipc.telstra.net) has joined #ceph
[0:10] * derjohn_mob (~aj@x590e0aa0.dyn.telefonica.de) Quit (Ping timeout: 480 seconds)
[0:11] * luckz (~Eman@26XAAADXS.tor-irc.dnsbl.oftc.net) Quit ()
[0:11] * chrisinajar (~theghost9@cry.ip-eend.nl) has joined #ceph
[0:28] * ronrib (~boswortr@45.32.242.135) Quit (Ping timeout: 480 seconds)
[0:41] * chrisinajar (~theghost9@61TAAANE4.tor-irc.dnsbl.oftc.net) Quit ()
[0:41] * anadrom (~Tarazed@217.23.14.190) has joined #ceph
[0:43] * willi (~willi@p200300774E0F3CFC4499B5C5E8D91510.dip0.t-ipconnect.de) Quit (Remote host closed the connection)
[0:51] * cholcombe (~chris@c-73-180-29-35.hsd1.or.comcast.net) has joined #ceph
[0:55] * ntpttr_ (~ntpttr@192.55.54.42) has joined #ceph
[0:57] * ChrisHolcombe (~chris@c-73-180-29-35.hsd1.or.comcast.net) Quit (Ping timeout: 480 seconds)
[1:00] * ntpttr (~ntpttr@192.55.54.42) Quit (Ping timeout: 480 seconds)
[1:03] * anadrom (~Tarazed@26XAAADZH.tor-irc.dnsbl.oftc.net) Quit (Ping timeout: 480 seconds)
[1:06] * rendar (~I@host178-143-dynamic.59-82-r.retail.telecomitalia.it) Quit (Quit: std::lower_bound + std::less_equal *works* with a vector without duplicates!)
[1:13] * oms101 (~oms101@p20030057EA5BFD00C6D987FFFE4339A1.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[1:15] * penguinRaider__ (~KiKo@103.6.219.219) Quit (Ping timeout: 480 seconds)
[1:16] * stiopa (~stiopa@cpc73832-dals21-2-0-cust453.20-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[1:20] * ntpttr___ (~ntpttr@134.134.139.77) has joined #ceph
[1:22] * oms101 (~oms101@p20030057EA48B800C6D987FFFE4339A1.dip0.t-ipconnect.de) has joined #ceph
[1:24] * penguinRaider__ (~KiKo@103.6.219.219) has joined #ceph
[1:25] * ntpttr_ (~ntpttr@192.55.54.42) Quit (Ping timeout: 480 seconds)
[1:29] * kuku (~kuku@119.93.91.136) has joined #ceph
[1:34] * SweetGirl (~isaxi@62-210-37-82.rev.poneytelecom.eu) has joined #ceph
[1:35] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) has joined #ceph
[1:37] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) Quit ()
[1:38] * neurodrone_ (~neurodron@162.243.191.67) has joined #ceph
[1:44] * willi (~willi@p200300774E0F3CFC4499B5C5E8D91510.dip0.t-ipconnect.de) has joined #ceph
[1:51] * truan-wang (~truanwang@58.247.8.186) has joined #ceph
[1:52] * willi (~willi@p200300774E0F3CFC4499B5C5E8D91510.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[1:53] * toastydeath (~toast@pool-71-255-253-39.washdc.fios.verizon.net) has joined #ceph
[1:54] * theTrav (~theTrav@ipc032.ipc.telstra.net) Quit (Remote host closed the connection)
[1:54] * jermudgeon (~jhaustin@southend.mdu.whitestone.link) Quit (Quit: jermudgeon)
[1:54] * jermudgeon (~jhaustin@southend.mdu.whitestone.link) has joined #ceph
[2:03] * SweetGirl (~isaxi@26XAAAD0P.tor-irc.dnsbl.oftc.net) Quit ()
[2:04] * Tarazed (~Diablothe@atlantic480.us.unmetered.com) has joined #ceph
[2:06] * debian112 (~bcolbert@207.183.247.46) has joined #ceph
[2:13] * danieagle (~Daniel@189-46-61-76.dsl.telesp.net.br) Quit (Quit: Obrigado por Tudo! :-) inte+ :-))
[2:16] <TheSov> soon we will all need ceph clusters at home to hold the contents of bitcoin's blockchain
[2:19] * EinstCrazy (~EinstCraz@180.174.49.62) has joined #ceph
[2:20] * EinstCrazy (~EinstCraz@180.174.49.62) Quit (Remote host closed the connection)
[2:26] * ronrib (~boswortr@45.32.242.135) has joined #ceph
[2:33] * Tarazed (~Diablothe@61TAAANHN.tor-irc.dnsbl.oftc.net) Quit ()
[2:37] * Jeffrey4l_ (~Jeffrey@119.251.244.121) has joined #ceph
[2:38] * redbeast12 (~Thayli@0x667.crypt.gy) has joined #ceph
[2:38] * valeech (~valeech@pool-108-44-162-111.clppva.fios.verizon.net) Quit (Quit: valeech)
[2:46] * borei (~dan@node-1w7jr9qle4x5ix2kjybp8d4fv.ipv6.telus.net) Quit (Quit: Leaving.)
[2:47] * aarontc (~aarontc@2001:470:e893::1:1) Quit (Quit: Bye!)
[2:49] * praveen (~praveen@122.172.150.27) Quit (Read error: Connection reset by peer)
[2:55] * aarontc (~aarontc@2001:470:e893::1:1) has joined #ceph
[2:56] * praveen (~praveen@122.172.150.27) has joined #ceph
[3:00] * praveen (~praveen@122.172.150.27) Quit (Read error: Connection reset by peer)
[3:04] * praveen (~praveen@122.172.150.27) has joined #ceph
[3:08] * redbeast12 (~Thayli@5AEAAADKU.tor-irc.dnsbl.oftc.net) Quit ()
[3:08] * lmg (~kiasyn@213.61.149.100) has joined #ceph
[3:13] * praveen (~praveen@122.172.150.27) Quit (Ping timeout: 480 seconds)
[3:38] * lmg (~kiasyn@26XAAAD21.tor-irc.dnsbl.oftc.net) Quit ()
[3:38] * Inverness (~Catsceo@185.65.134.75) has joined #ceph
[3:49] * EinstCrazy (~EinstCraz@203.79.187.188) has joined #ceph
[3:56] * yanzheng (~zhyan@125.70.23.222) has joined #ceph
[4:08] * Inverness (~Catsceo@26XAAAD3R.tor-irc.dnsbl.oftc.net) Quit ()
[4:08] * darkid (~Grimhound@tor-exit.ohdoom.net) has joined #ceph
[4:09] * kuku (~kuku@119.93.91.136) Quit (Remote host closed the connection)
[4:14] * neurodrone_ (~neurodron@162.243.191.67) Quit (Ping timeout: 480 seconds)
[4:19] * dnunez (~dnunez@c-73-38-0-185.hsd1.ma.comcast.net) Quit (Quit: Leaving)
[4:22] <ceph-ircslackbot1> <blairo> hi all. we???re doing a Hammer->Jewel upgrade at the moment and getting some horrible slow request issues when bringing OSDs back up. they are clearing on their own but seeing almost 5 mins of blockage in some cases
[4:22] <ceph-ircslackbot1> <blairo> anyone else seen this?
[4:22] <ceph-ircslackbot1> <blairo> seems like it might be related to having the OSDs down for a while whilst we???re chown-ing the /var/lib/ceph/??? OSD directories to the new ???ceph??? uid
[4:23] * flisky (~Thunderbi@106.38.61.189) has joined #ceph
[4:33] * vicente (~~vicente@125-227-238-55.HINET-IP.hinet.net) has joined #ceph
[4:36] * jermudgeon (~jhaustin@southend.mdu.whitestone.link) Quit (Read error: Connection reset by peer)
[4:38] * darkid (~Grimhound@61TAAANKO.tor-irc.dnsbl.oftc.net) Quit ()
[4:38] * Grimmer (~lmg@159.148.186.194) has joined #ceph
[4:38] * lightspeed (~lightspee@2001:8b0:16e:1:8326:6f70:89f:8f9c) Quit (Ping timeout: 480 seconds)
[4:39] * jermudgeon (~jhaustin@southend.mdu.whitestone.link) has joined #ceph
[4:43] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) has joined #ceph
[4:49] * penguinRaider__ (~KiKo@103.6.219.219) Quit (Ping timeout: 480 seconds)
[4:58] * penguinRaider__ (~KiKo@103.6.219.219) has joined #ceph
[5:01] * kuku (~kuku@119.93.91.136) has joined #ceph
[5:08] * Grimmer (~lmg@26XAAAD4W.tor-irc.dnsbl.oftc.net) Quit ()
[5:08] * colde1 (~redbeast1@104.ip-167-114-238.eu) has joined #ceph
[5:09] * Vacuum_ (~Vacuum@i59F791AA.versanet.de) has joined #ceph
[5:16] * Vacuum__ (~Vacuum@88.130.203.172) Quit (Ping timeout: 480 seconds)
[5:16] * jermudgeon (~jhaustin@southend.mdu.whitestone.link) Quit (Read error: No route to host)
[5:17] * jermudgeon (~jhaustin@31.207.56.59) has joined #ceph
[5:23] <ron-slc> blario : The longer an OSD is down, likely the more it will need to have sent in updated bits. Maybe it would help to set --osd-max-backfills 1, and --osd-max-recovery-threads 1.
[5:24] <ron-slc> There is a mailing archive I used in the past, for this. Google search for: "ceph Backfill and Recovery traffic shaping"
[5:24] <ron-slc> link:http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-April/038989.html
[5:25] <ron-slc> PS. I'm still on Hammer, on production. I have only lifted Testing/Dev to Infernalis
[5:28] <ron-slc> Blario: also check that deep-scrubs are disabled during your migration. ceph osd set nodeep-scrub
[5:29] <ron-slc> MAKE SUPER SURE YOU RE_ENABLE THEM AGAIN when done: ceph osd unset nodeep-scrub
[5:30] <ron-slc> I have a CRON, which disables deep-scrub between 8AM and 6PM, on weekdays. So all this busy work is done when less people are on the systems.
[5:31] * jermudgeon (~jhaustin@31.207.56.59) Quit (Quit: jermudgeon)
[5:31] * flisky (~Thunderbi@106.38.61.189) Quit (Quit: flisky)
[5:34] * jermudgeon (~jhaustin@southend.mdu.whitestone.link) has joined #ceph
[5:38] * colde1 (~redbeast1@9YSAAAOGQ.tor-irc.dnsbl.oftc.net) Quit ()
[5:41] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) Quit (Quit: neurodrone_)
[5:42] * sudocat (~dibarra@104-188-116-197.lightspeed.hstntx.sbcglobal.net) has joined #ceph
[5:44] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) has joined #ceph
[5:48] * ffilzwin2 (~ffilz@c-76-115-190-27.hsd1.or.comcast.net) Quit (Quit: Leaving)
[5:48] * praveen (~praveen@121.244.155.8) has joined #ceph
[5:48] * praveen (~praveen@121.244.155.8) Quit (Remote host closed the connection)
[5:48] * praveen (~praveen@121.244.155.8) has joined #ceph
[5:48] * vimal (~vikumar@114.143.165.70) has joined #ceph
[5:50] * sudocat (~dibarra@104-188-116-197.lightspeed.hstntx.sbcglobal.net) Quit (Ping timeout: 480 seconds)
[5:51] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[5:51] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) Quit (Quit: neurodrone_)
[5:55] * truan-wang (~truanwang@58.247.8.186) Quit (Ping timeout: 480 seconds)
[6:01] * sudocat (~dibarra@192.185.1.20) has joined #ceph
[6:03] * IvanJobs (~ivanjobs@103.50.11.146) has joined #ceph
[6:05] * truan-wang (~truanwang@220.248.17.34) has joined #ceph
[6:07] * vimal (~vikumar@114.143.165.70) Quit (Quit: Leaving)
[6:14] * truan-wang (~truanwang@220.248.17.34) Quit (Ping timeout: 480 seconds)
[6:21] * ffilzwin (~ffilz@c-76-115-190-27.hsd1.or.comcast.net) has joined #ceph
[6:23] * penguinRaider__ (~KiKo@103.6.219.219) Quit (Ping timeout: 480 seconds)
[6:24] * theTrav (~theTrav@203.35.9.142) has joined #ceph
[6:25] * IvanJobs (~ivanjobs@103.50.11.146) Quit (Read error: Connection reset by peer)
[6:25] <theTrav> so I've ls
[6:26] <theTrav> so I've got ceph-deploy installing a ceph cluster
[6:26] * IvanJobs (~ivanjobs@122.14.140.7) has joined #ceph
[6:26] <theTrav> however http://docs.ceph.com/docs/master/install/install-ceph-gateway/ says it should have a rgw command
[6:26] <theTrav> it disagrees
[6:26] <theTrav> ceph-deploy: error: argument COMMAND: invalid choice: 'rgw' (choose from 'new', 'install', 'mds', 'mon', 'gatherkeys', 'disk', 'osd', 'admin', 'config', 'uninstall', 'purgedata', 'purge', 'forgetkeys', 'pkg')
[6:26] <theTrav> ceph-deploy v 1.4.0
[6:27] <tserong> that's very old
[6:28] <tserong> like, ~march 2014
[6:28] * vimal (~vikumar@121.244.87.116) has joined #ceph
[6:28] <theTrav> the version or the instructions?
[6:28] <tserong> the ceph-deploy version
[6:28] <theTrav> what is latest stable?
[6:29] <tserong> v1.5.34, juding from the tags in the source repo
[6:30] * sudocat (~dibarra@192.185.1.20) Quit (Ping timeout: 480 seconds)
[6:30] <theTrav> http://docs.ceph.com/docs/jewel/start/quick-start-preflight/ <- we followed that
[6:30] <tserong> looks like the rgw support has been in ceph-deploy since v1.5.23
[6:31] * nils_ (~nils_@doomstreet.collins.kg) has joined #ceph
[6:31] <tserong> where did the version of ceph-deploy you're using come from?
[6:35] * IvanJobs (~ivanjobs@122.14.140.7) Quit (Read error: Connection reset by peer)
[6:35] * IvanJobs (~ivanjobs@103.50.11.146) has joined #ceph
[6:40] * praveen (~praveen@121.244.155.8) Quit ()
[6:41] <theTrav> ok, so I found it
[6:41] <theTrav> sorry, our bad
[6:41] <tserong> np
[6:41] <theTrav> 1.4.0 is the version sitting in 14.04LTS apt repo
[6:42] <theTrav> So we didn't add the ceph ppa or equivalent
[6:43] * utugi______ (~FierceFor@46.182.106.190) has joined #ceph
[6:47] <tserong> ah. you might want the packages from http://download.ceph.com/debian-jewel/ or http://download.ceph.com/debian/ -- there's some notes about this at http://docs.ceph.com/docs/master/install/get-packages/
[6:47] <tserong> (i haven't tried an ubuntu install for ages though)(
[6:48] <theTrav> yeah we found that
[6:48] <theTrav> in fact, when we first installed we used that
[6:48] <theTrav> however when we encoded our process in some scripts we must have missed that line
[6:48] <theTrav> I'm saying we... I should be blaming someone else here
[6:49] <tserong> :)
[6:49] <theTrav> it was _that guy_
[6:49] <tserong> man, i hate that guy
[6:49] <tserong> ;)
[6:51] <theTrav> inorite!
[7:02] * penguinRaider (~KiKo@103.6.219.219) has joined #ceph
[7:02] * truan-wang (~truanwang@220.248.17.34) has joined #ceph
[7:06] * dis (~dis@00018d20.user.oftc.net) Quit (Ping timeout: 480 seconds)
[7:06] * lkoranda (~lkoranda@nat-pool-brq-t.redhat.com) Quit (Ping timeout: 480 seconds)
[7:07] * lkoranda (~lkoranda@149.11.118.10) has joined #ceph
[7:12] * kawa2014 (~kawa@dynamic-adsl-84-221-91-70.clienti.tiscali.it) has joined #ceph
[7:13] * utugi______ (~FierceFor@26XAAAD7K.tor-irc.dnsbl.oftc.net) Quit ()
[7:13] * adept256 (~slowriot@tor2e1.privacyfoundation.ch) has joined #ceph
[7:17] * penguinRaider (~KiKo@103.6.219.219) Quit (Ping timeout: 480 seconds)
[7:24] * vikhyat (~vumrao@121.244.87.116) has joined #ceph
[7:27] * penguinRaider (~KiKo@103.6.219.219) has joined #ceph
[7:31] * karnan (~karnan@121.244.87.117) has joined #ceph
[7:37] * rdas (~rdas@121.244.87.116) has joined #ceph
[7:41] * EinstCra_ (~EinstCraz@58.247.119.250) has joined #ceph
[7:41] * rotbeard (~redbeard@aftr-109-90-233-215.unity-media.net) has joined #ceph
[7:43] * adept256 (~slowriot@5AEAAADQV.tor-irc.dnsbl.oftc.net) Quit ()
[7:43] * Hejt (~VampiricP@2.tor.exit.babylon.network) has joined #ceph
[7:44] * EinstCr__ (~EinstCraz@203.79.187.188) has joined #ceph
[7:44] * EinstC___ (~EinstCraz@58.247.119.250) has joined #ceph
[7:44] * EinstCra_ (~EinstCraz@58.247.119.250) Quit (Read error: No route to host)
[7:45] * derjohn_mob (~aj@x590e62fa.dyn.telefonica.de) has joined #ceph
[7:48] * EinstCrazy (~EinstCraz@203.79.187.188) Quit (Ping timeout: 480 seconds)
[7:48] * ceph-ircslackbot1 (~ceph-ircs@ds9536.dreamservers.com) Quit (Remote host closed the connection)
[7:48] * ceph-ircslackbot (~ceph-ircs@ds9536.dreamservers.com) has joined #ceph
[7:49] * willi (~willi@p5797BF0B.dip0.t-ipconnect.de) has joined #ceph
[7:49] * xarses_ (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) Quit (Remote host closed the connection)
[7:50] * xarses (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) has joined #ceph
[7:52] * EinstCr__ (~EinstCraz@203.79.187.188) Quit (Ping timeout: 480 seconds)
[7:57] * derjohn_mob (~aj@x590e62fa.dyn.telefonica.de) Quit (Ping timeout: 480 seconds)
[7:57] * willi (~willi@p5797BF0B.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[8:01] * derjohn_mob (~aj@x590e62fa.dyn.telefonica.de) has joined #ceph
[8:04] * rdas (~rdas@121.244.87.116) Quit (Quit: Leaving)
[8:09] * swami1 (~swami@49.38.0.153) has joined #ceph
[8:11] * rakeshgm (~rakesh@121.244.87.117) has joined #ceph
[8:13] * Hejt (~VampiricP@9YSAAAOJ6.tor-irc.dnsbl.oftc.net) Quit ()
[8:13] * Misacorp (~Arfed@93.174.90.30) has joined #ceph
[8:14] * TMM (~hp@178-84-46-106.dynamic.upc.nl) Quit (Quit: Ex-Chat)
[8:28] * rdas (~rdas@121.244.87.116) has joined #ceph
[8:31] * arcimboldo (~antonio@84-75-174-248.dclient.hispeed.ch) has joined #ceph
[8:37] * kmajk (~kmajk@nat-hq.ext.getresponse.com) has joined #ceph
[8:39] * sankarshan (~sankarsha@ip-64-134-64-4.public.wayport.net) has joined #ceph
[8:43] * Misacorp (~Arfed@61TAAANPY.tor-irc.dnsbl.oftc.net) Quit ()
[8:45] * dis (~dis@00018d20.user.oftc.net) has joined #ceph
[8:49] * lkoranda (~lkoranda@149.11.118.10) Quit (Ping timeout: 480 seconds)
[8:49] * lkoranda (~lkoranda@nat-pool-brq-t.redhat.com) has joined #ceph
[8:58] * sickolog1 (~mio@vpn.bcs.hr) has joined #ceph
[8:58] * sickology (~mio@vpn.bcs.hr) Quit (Read error: Connection reset by peer)
[9:00] * rendar (~I@host1-139-dynamic.49-82-r.retail.telecomitalia.it) has joined #ceph
[9:03] * TMM (~hp@185.5.121.201) has joined #ceph
[9:06] * Kurt (~Adium@2001:628:1:5:104:2704:e8c9:18b9) has joined #ceph
[9:06] * willi (~willi@2a00:1050:4:0:406a:e640:4a9b:a3fc) has joined #ceph
[9:08] * willi (~willi@2a00:1050:4:0:406a:e640:4a9b:a3fc) Quit ()
[9:09] * willi (~willi@212.124.32.5) has joined #ceph
[9:10] * allaok (~allaok@machine107.orange-labs.com) has joined #ceph
[9:13] * Jase (~tritonx@158.69.194.36) has joined #ceph
[9:13] * sickolog1 (~mio@vpn.bcs.hr) Quit (Read error: Connection reset by peer)
[9:13] * sickology (~mio@vpn.bcs.hr) has joined #ceph
[9:20] * boolman (boolman@79.138.78.238) has joined #ceph
[9:21] * sep (~sep@95.62-50-191.enivest.net) has joined #ceph
[9:21] * Jeffrey4l_ (~Jeffrey@119.251.244.121) Quit (Ping timeout: 480 seconds)
[9:22] * derjohn_mob (~aj@x590e62fa.dyn.telefonica.de) Quit (Ping timeout: 480 seconds)
[9:23] * Jeffrey4l_ (~Jeffrey@119.251.244.121) has joined #ceph
[9:27] * EinstC___ (~EinstCraz@58.247.119.250) Quit (Read error: Connection reset by peer)
[9:27] * EinstCrazy (~EinstCraz@58.247.119.250) has joined #ceph
[9:27] <TMM> I think one of my OSDs got corrupted because I have a drive writing garbage, it seems that this corruption spread across that entire PG though, is that possible?
[9:32] <willi> hey guys after a ceph-deploy config push nodes...
[9:33] <willi> how can i activate the new config on all nodes without a OS reboot
[9:33] <willi> ?
[9:36] * penguinRaider (~KiKo@103.6.219.219) Quit (Ping timeout: 480 seconds)
[9:40] <willi> no one ?
[9:43] * Jase (~tritonx@61TAAANRO.tor-irc.dnsbl.oftc.net) Quit ()
[9:43] * osuka_ (~blip2@exit.tor.uwaterloo.ca) has joined #ceph
[9:43] <boolman> eiter restart the affected daemons or injargs I assume
[9:44] <boolman> injectargs even
[9:44] * rraja (~rraja@121.244.87.117) has joined #ceph
[9:45] * penguinRaider (~KiKo@103.6.219.219) has joined #ceph
[9:47] * jamespag` is now known as jamespage
[9:48] * mashwo00 (~textual@51.179.162.234) has joined #ceph
[9:51] * mivaho (~quassel@2001:983:eeb4:1:c0de:69ff:fe2f:5599) Quit (Quit: Going)
[9:51] * mivaho (~quassel@2001:983:eeb4:1:c0de:69ff:fe2f:5599) has joined #ceph
[9:51] * sankarshan (~sankarsha@ip-64-134-64-4.public.wayport.net) Quit (Quit: Are you sure you want to quit this channel (Cancel/Ok) ?)
[9:57] * fsimonce (~simon@host99-64-dynamic.27-79-r.retail.telecomitalia.it) has joined #ceph
[10:03] * kuku (~kuku@119.93.91.136) Quit (Remote host closed the connection)
[10:03] * DanFoster (~Daniel@2a00:1ee0:3:1337:70ba:b487:2b70:d2e3) has joined #ceph
[10:06] * karnan (~karnan@121.244.87.117) Quit (Ping timeout: 480 seconds)
[10:08] * b0e (~aledermue@213.95.25.82) has joined #ceph
[10:12] * bara (~bara@nat-pool-brq-t.redhat.com) has joined #ceph
[10:13] * osuka_ (~blip2@61TAAANSA.tor-irc.dnsbl.oftc.net) Quit ()
[10:13] * theghost99 (~neobenedi@tor1.mysec-arch.net) has joined #ceph
[10:16] * karnan (~karnan@121.244.87.117) has joined #ceph
[10:20] * georgem (~Adium@85.204.4.209) has joined #ceph
[10:25] * derjohn_mob (~aj@2001:6f8:1337:0:b56f:1c5c:41a8:95c1) has joined #ceph
[10:32] * theTrav (~theTrav@203.35.9.142) Quit (Remote host closed the connection)
[10:35] * ngoswami (~ngoswami@1.39.87.139) has joined #ceph
[10:39] * briner (~briner@2001:620:600:1000:5d26:8eaa:97f0:8115) has joined #ceph
[10:41] * dyasny (~dyasny@cable-192.222.152.136.electronicbox.net) Quit (Ping timeout: 480 seconds)
[10:41] * maybebuggy (~maybebugg@2a01:4f8:191:2350::2) has joined #ceph
[10:42] <maybebuggy> Hi, quick question - is there a way to get a "histogram" of ceph client io response times? or at least have an idea how many requests are "blocked" longer than 1s, 5s, 10s?
[10:43] * theghost99 (~neobenedi@61TAAANS0.tor-irc.dnsbl.oftc.net) Quit ()
[10:43] * w2k (~TomyLobo@torrelay6.tomhek.net) has joined #ceph
[10:43] * theTrav (~theTrav@1.152.97.120) has joined #ceph
[10:44] * dyasny (~dyasny@cable-192.222.152.136.electronicbox.net) has joined #ceph
[10:45] * branto (~branto@ip-78-102-208-181.net.upcbroadband.cz) has joined #ceph
[10:45] * jermudgeon (~jhaustin@southend.mdu.whitestone.link) Quit (Quit: jermudgeon)
[10:55] <etienneme> You can search on logs
[10:55] <etienneme> you will find slow/blocked requests
[10:55] * truan-wang (~truanwang@220.248.17.34) Quit (Remote host closed the connection)
[10:56] * dgurtner (~dgurtner@178.197.227.242) has joined #ceph
[10:56] <maybebuggy> etienneme: but only the ones > 32s - right?
[10:57] <willi> hi there: crush map has legacy tunables (require bobtail, min is firefly)
[10:57] <willi> what does it mean on jewel?
[10:57] <willi> i have set ceph osd crush tunables bobtail
[10:57] <willi> because i am using ubuntu 14.04 with 3.13 kernel
[10:57] <willi> everything correct ??
[10:58] <etienneme> maybebuggy: I don't really know what "blocked" means, maybe it's jsut slower than 32 seconds.
[10:58] <maybebuggy> etienneme: yes, its logs like that: "ceph-osd.23.log:2016-07-18 10:42:54.797848 7f1430984700 0 log_channel(cluster) log [WRN] : slow request 34.780955 seconds old, received at 2016-07-18 10:42:20.015699: osd_op(client.50711839.0:1082561 rbd_data.305633c10937fc9.0000000000000784 [stat,set-alloc-hint object_size 8388608 write_size 8388608,write 606208~4096] 11.5ffda6e3 ack+ondisk+write+known_if_redirected e2572) currently waiting for rw locks"
[10:58] <etienneme> willi: Where have you found this?
[10:59] <willi> http://docs.ceph.com/docs/master/rados/operations/crush-map/
[10:59] <willi> Which client versions support CRUSH_TUNABLES2
[10:59] <willi> v0.55 or later, including bobtail series (v0.56.x)
[10:59] <willi> Linux kernel version v3.9 or later (for the file system and RBD kernel clients)
[10:59] <maybebuggy> that makes ceph go into "WARN" state when above 32s. But I'd like to also see requests > 1s...
[10:59] <etienneme> slow requests do not answer your question?
[11:00] <etienneme> willi: Your ubuntu 14.04 3.13 kernel is server or client?
[11:00] <willi> both
[11:01] <etienneme> tunable only affects client, with this old kernel you can only use bobtail
[11:01] <maybebuggy> etienneme: they only get printed for requests > 32s - not for > 1s
[11:02] <willi> okay
[11:02] <willi> is it a problem the warning ?
[11:02] <maybebuggy> i had hoped there might be some stat / perf counter which could be used for that
[11:02] <etienneme> I have logs like this "cluster [WRN] 1 slow requests, 1 included below; oldest blocked for > 30.047272 secs"
[11:02] <etienneme> I don't know :(
[11:02] * georgem (~Adium@85.204.4.209) Quit (Quit: Leaving.)
[11:02] <etienneme> willi: no it's just bad to use an old tunable
[11:03] <willi> okay
[11:04] <maybebuggy> etienneme: yes, but only for requests which take more than 32 secs. nothing for the amount of requests between 1s and 32s :)
[11:05] <etienneme> You'll have to monitor :p
[11:07] <maybebuggy> i can't - besides setting the complaint time to a small number which means the cluster is ALWAYS in state warn
[11:07] <maybebuggy> not really desireable, i'd say
[11:09] * theTrav (~theTrav@1.152.97.120) Quit (Remote host closed the connection)
[11:10] <willi> guys: who of you have an test cluster? i think i have found a bug in jewel. we have 18 data servers over 3 racks. crushmap=rack. tested with ubuntu 14.04 / 16.04 and debian jessie. if we shut down a whole rack ceph becomes inaccessible f??r the client. rbd ls on the client gives us no feedback. after 900 seconds cluster ran into pg_stats timeout and the client can again access the cluster. with ceph infernalis NO PROBLEM !
[11:11] <willi> is anyone here who can test this scenario ?
[11:13] * w2k (~TomyLobo@5AEAAADVM.tor-irc.dnsbl.oftc.net) Quit ()
[11:14] * fdmanana (~fdmanana@2001:8a0:6e0c:6601:7981:6bc4:cb04:3446) has joined #ceph
[11:17] <willi> no one??
[11:17] * biGGer (~Swompie`@195-154-15-227.rev.poneytelecom.eu) has joined #ceph
[11:17] <willi> what is going ? is no one of you doing availability test?
[11:17] * truan-wang (~truanwang@112.65.190.56) has joined #ceph
[11:19] * penguinRaider (~KiKo@103.6.219.219) Quit (Ping timeout: 480 seconds)
[11:27] <maybebuggy> willi: maybe the ones doing those are not online at the moment? also that seems more like a devel question so maybe the relevant people might be in #ceph-devel
[11:31] * kawa2014 (~kawa@dynamic-adsl-84-221-91-70.clienti.tiscali.it) Quit (Quit: Leaving)
[11:33] * penguinRaider (~KiKo@103.6.219.219) has joined #ceph
[11:34] * truan-wang (~truanwang@112.65.190.56) Quit (Ping timeout: 480 seconds)
[11:44] * penguinRaider (~KiKo@103.6.219.219) Quit (Ping timeout: 480 seconds)
[11:47] * biGGer (~Swompie`@5AEAAADWJ.tor-irc.dnsbl.oftc.net) Quit ()
[11:47] * Xa (~FierceFor@relay1.tor.openinternet.io) has joined #ceph
[11:58] * dan__ (~Daniel@2a00:1ee0:3:1337:a1cf:53f2:472a:9931) has joined #ceph
[11:58] * kawa2014 (~kawa@89.184.114.246) has joined #ceph
[12:00] * penguinRaider (~KiKo@103.6.219.219) has joined #ceph
[12:05] * DanFoster (~Daniel@2a00:1ee0:3:1337:70ba:b487:2b70:d2e3) Quit (Ping timeout: 480 seconds)
[12:08] * TMM (~hp@185.5.121.201) Quit (Remote host closed the connection)
[12:13] <brians_> drop everything quick! willi needs us to test a scenario. I'll go build a test cluster for you now, how many nodes?
[12:16] * truan-wang (~truanwang@58.247.8.186) has joined #ceph
[12:17] * Xa (~FierceFor@26XAAAEED.tor-irc.dnsbl.oftc.net) Quit ()
[12:17] * Defaultti1 (~Gecko1986@26XAAAEFI.tor-irc.dnsbl.oftc.net) has joined #ceph
[12:17] <willi> i have 18 nodes
[12:18] <willi> 6 per rack
[12:19] <willi> should i give you my config?
[12:19] <willi> can you use ubuntu 14.04 an jewel for the test?
[12:20] <willi> plus 3 mon nodes
[12:20] <willi> plus 3 client nodes
[12:21] <willi> mon 1 in rack1, client 1 in rack 1, data node1-6 in rack1, mon 2 in rack2, client 2 in rack 2, data node 7-12 in rack2, and so on....
[12:23] <sep> willi, what is your size and min_size ? with 6 node racks you only have 3 racks. and that is the default size. so if you lost one you would be degraded. i'd reccomend have $size + how-many-you-want-to-tolerate-loosing number of racks.
[12:24] <willi> ceph osd pool set rbd min_size 1
[12:24] <willi> ceph osd pool set rbd size 3
[12:25] <willi> i can give you me test feedback in ca. 1 hour infernalis is just installing on my cluster
[12:25] <willi> i bet that the problem is than gone away when i shut down a whole rack
[12:26] * TMM (~hp@185.5.121.201) has joined #ceph
[12:27] * arcimboldo (~antonio@84-75-174-248.dclient.hispeed.ch) Quit (Ping timeout: 480 seconds)
[12:28] * hybrid512 (~walid@195.200.189.206) has joined #ceph
[12:28] <willi> each server has 5 osd's plus 2 ssd's for journal
[12:29] <willi> so 90 osd in the whole cluster plus 36 ssd's
[12:29] <willi> pg is at 4096
[12:30] <willi> sep, you mean i should set ceph osd pool set rbd size 4 ??
[12:30] * truan-wang (~truanwang@58.247.8.186) Quit (Ping timeout: 480 seconds)
[12:32] * penguinRaider (~KiKo@103.6.219.219) Quit (Ping timeout: 480 seconds)
[12:33] <willi> brians_, which information do you need additionally ?
[12:35] * dgurtner (~dgurtner@178.197.227.242) Quit (Read error: No route to host)
[12:39] <sep> willi, no. with size 3 you want to have 3 copies. spread out over your crushmap, since you use rack as your failure domain you end up with 1 of the 3 copies on each rack
[12:41] <sep> willi, now when you loose a rack. then you are degraded, and i think that's bad. ceph tries to copy a opbject on the lost rack to a new location, to maintain size 3. but there is no place to put it. since it would be on the same failure domain as the other 2 objects.
[12:41] <sep> hence if you want to loose 1 rack with size 3, you want 4 racks.
[12:41] <willi> http://pastebin.com/3LWefLHN
[12:41] <sep> if you want to be able to loose 2 racks you want 5 in total
[12:42] * penguinRaider (~KiKo@103.6.219.219) has joined #ceph
[12:42] <sep> i am no expert tho. but i think running degraded is a bad situation that you are supposed to avoid.
[12:43] <willi> so you mean i must set: ceph osd pool set rbd size 4???
[12:43] <sep> no
[12:43] <sep> i mean you must have 1 more rack then you have size set
[12:43] <willi> you mean ceph osd pool set rbd size 3 and i must change the crush map to 4 racks ?
[12:44] <willi> okay but than how get the data distributed if i have only 3 copies ?
[12:44] <willi> 3 copies over 4 racks?
[12:44] <sep> yes. there are many many objects. so the 3 copies will spread out
[12:45] <willi> i can test it
[12:45] <willi> i put 4 servers per rack
[12:45] <willi> so i have 4 racks
[12:45] <willi> with 16 nodes
[12:46] <willi> but before i test it again with infernalis
[12:47] <willi> i thougt that if i have 3 racks und ceph osd pool set rbd size 3 = copy 1 goes to rack1 copy 2 goes to rack2 and copy 3 goes to rack 3
[12:47] * Defaultti1 (~Gecko1986@26XAAAEFI.tor-irc.dnsbl.oftc.net) Quit ()
[12:47] <willi> and striped in the rack over 6 nodes
[12:48] <willi> not right?
[12:50] <sep> no they will go to 1 of the nodes in the rack.
[12:50] <willi> 3 copies to only 1 node?
[12:50] <willi> can't be!!!
[12:50] <willi> then you have data loss
[12:50] <willi> if this node goes down
[12:51] * _28_ria (~kvirc@opfr028.ru) Quit (Read error: Connection reset by peer)
[12:51] <sep> one of the 3 copies gotes to one of the nodes in each rack. total 3 copies
[12:51] <willi> yes
[12:52] <willi> copy 1 in rack1, copy 2 in rack2, copy 3 in rack3
[12:52] <sep> but if you loose a rack. there is no free space to put copy nr 3. since you only have 3 racks. if you had 4 racks. ceph could put the object on the rack that did not previously have a copy of that object
[12:53] <willi> ah okay
[12:53] <sep> so with 3 out of 3 racks you have no free space to put objects on = degraded. if you have 3 out of 4, you have 1 rack to put objects on, but you need available space to actualy DO it when a rack dies. so you can not fill more then 66% . with size 3 out of 10 racks you can fill a lot more since a object have 6 others racks to pick from when a rack dies.
[12:54] <willi> so that i am not in degraded mode
[12:54] <sep> that's also how you get aggeregate performance by spreading the load of the RBD inmage out over more machines
[12:54] <sep> willi, correct
[12:55] <willi> so if a decrease: ceph osd pool set rbd size 3
[12:55] <willi> to : ceph osd pool set rbd size 2
[12:56] <willi> and with my 3 racks
[12:56] <willi> i have that what i want
[12:56] <sep> yes. but there i a reason why the default is 3.
[12:56] <willi> and: ceph osd pool set rbd min_size 1
[12:56] <willi> hmm okay and what is the reason?
[12:56] <sep> if a object is degraded on disk. or somewhere in the path. with 2 objects it's hard to know witch is correct.
[12:56] <sep> with 3, one can compare
[12:57] <willi> okay
[12:57] <willi> okay but say i have my cluster only 3 racks...
[12:57] <willi> 3 copies...
[12:57] * dgurtner (~dgurtner@178.197.227.242) has joined #ceph
[12:57] <willi> i don't want to fill 66%
[12:58] <willi> you can see here
[12:58] <willi> https://paste.ee/p/BtUN2
[12:58] * arcimboldo (~antonio@dhcp-y11-zi-s3it-130-60-34-054.uzh.ch) has joined #ceph
[12:58] <willi> 366 GB data, 1135 GB used, 82656 GB
[12:58] <kmajk> willi: rbd size 3 and osd crush chooseleaf type = 3 , so 3 copies on 3 nodes in different racks
[12:58] <willi> most people told me type = 0
[12:59] <willi> do you have a documentation for that? type = 3
[12:59] <kmajk> yes
[12:59] <kmajk> http://docs.ceph.com/docs/master/rados/operations/crush-map/ types
[13:00] <kmajk> type = 0 is very bad for chooseleaf type becouse it's type OSD, so 3 copies could be on the same node
[13:01] <willi> ahh okay
[13:02] <willi> so and what do you mean about that what sep has said?
[13:02] <willi> should i use 4 racks with 3 copies?
[13:02] <kmajk> type = 1 is recomended if you have 3 host without rack abstraction , so 3 copies on 3 different hosts
[13:04] <kmajk> if you have type = 3 rack, and size 3 if one host in rack fail, it will rebalance to other host in this rack (if there are any), on other host on 4th rack
[13:05] <kmajk> ,on/or
[13:05] <willi> i want to kill a ceomplete rack !
[13:05] <willi> complete
[13:06] <kmajk> if you kill whole rack you mast have 4h rack for size 3 if not it will be degraded for ever
[13:06] <willi> and degraded means read only ?
[13:06] <kmajk> no
[13:06] <willi> okay
[13:06] <willi> i can live with degraded
[13:06] <kmajk> it will be r/w
[13:07] <willi> life sorry not live
[13:07] <willi> soo
[13:07] <kmajk> this is like in raid 1, if one disk fail you have working raid array
[13:07] <kmajk> but degraded
[13:07] <willi> but why stops my traffic on the rbd client ?
[13:07] <kmajk> until you insert next disk
[13:08] <willi> if i shut down a whole rack
[13:08] <willi> only in jewel
[13:08] <willi> with infernalis not
[13:08] <kmajk> and ofcours performance will be worse too
[13:08] <willi> yes thats okay
[13:08] <willi> i think about temporaliy power disruption for example....
[13:08] <sep> willi, do you have type=0 ? in that case you can have 3 copies on the same server in 1 rack on separate disks. or 3 copies on 3 servers in same rack
[13:09] <willi> yes i have type=0
[13:09] <kmajk> sep: it already told him that type = 0 is very bad
[13:09] <willi> i must change to type=3
[13:09] <sep> willi, type=0 is just for testing in developemnt or single machine configuration
[13:09] <sep> kmajk, indeed you have :)
[13:10] <sep> willi, yes and wait until it's done backfilling
[13:10] <kmajk> sep: or type=0 for backup solution :) for main cluster
[13:10] <willi> my infernalis is in ca. 30 minutes done
[13:10] <kmajk> sep: if you dont want 3nodes on offsite host
[13:10] <willi> than i test it with type=3
[13:10] <willi> after that
[13:11] <willi> i deinstall infernalis again and install jewel
[13:11] <willi> with type=3
[13:11] <willi> that we will see
[13:11] <willi> i paste that here
[13:11] <sep> kmajk, yes i have a single node cluster as well. for testing upgrades and changes on before trying on main one
[13:11] <willi> i am back in ca. 30-40 minutes
[13:11] * vicente (~~vicente@125-227-238-55.HINET-IP.hinet.net) Quit (Quit: Leaving)
[13:12] * wjw-freebsd (~wjw@smtp.digiware.nl) Quit (Ping timeout: 480 seconds)
[13:12] * Plord (~PKO@14.139.82.6) has joined #ceph
[13:13] * barteq (b@u.nix.club) has joined #ceph
[13:16] * penguinRaider (~KiKo@103.6.219.219) Quit (Ping timeout: 480 seconds)
[13:17] <The_Ball> i'm not sure how ceph knows how many objects the cluster has, hence my question, if a object is stored on three osds and those three were removed, would ceph show the objects as missing, or wouldn't you know the objects are missing
[13:17] * andrew_m (~Deiz@ns330209.ip-5-196-66.eu) has joined #ceph
[13:17] * alexxy (~alexxy@biod.pnpi.spb.ru) Quit (Quit: No Ping reply in 210 seconds.)
[13:18] * alexxy (~alexxy@biod.pnpi.spb.ru) has joined #ceph
[13:19] * willi (~willi@212.124.32.5) Quit (Ping timeout: 480 seconds)
[13:24] * johnavp19891 (~jpetrini@pool-100-14-10-2.phlapa.fios.verizon.net) Quit (Ping timeout: 480 seconds)
[13:27] * penguinRaider (~KiKo@103.6.219.219) has joined #ceph
[13:29] * bene2 (~bene@nat-pool-bos-t.redhat.com) has joined #ceph
[13:33] * willi (~willi@p200300774E3477FC642DA7132ED418F6.dip0.t-ipconnect.de) has joined #ceph
[13:43] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) has joined #ceph
[13:47] * andrew_m (~Deiz@9YSAAAOPA.tor-irc.dnsbl.oftc.net) Quit ()
[13:48] * ira (~ira@nat-pool-bos-u.redhat.com) has joined #ceph
[13:50] * kmajk (~kmajk@nat-hq.ext.getresponse.com) Quit (Quit: Lost terminal)
[13:51] * shyu (~Frank@218.241.172.114) has joined #ceph
[13:52] <sep> The_Ball, ... good question i am interested in the answer :)
[13:53] * Hemanth (~hkumar_@121.244.87.117) has joined #ceph
[13:54] * valeech (~valeech@pool-108-44-162-111.clppva.fios.verizon.net) has joined #ceph
[13:56] <willi> so guys i am back
[13:57] <willi> i give you in 5-10 minutes my test results
[13:58] * _28_ria (~kvirc@opfr028.ru) has joined #ceph
[14:02] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) Quit (Quit: neurodrone_)
[14:02] * lightspeed (~lightspee@2001:8b0:16e:1:8326:6f70:89f:8f9c) has joined #ceph
[14:06] * Racpatel (~Racpatel@2601:87:0:24af::1fbc) has joined #ceph
[14:07] * _28_ria (~kvirc@opfr028.ru) Quit (Read error: Connection reset by peer)
[14:09] * pdrakeweb (~pdrakeweb@oh-76-5-108-60.dhcp.embarqhsd.net) has joined #ceph
[14:10] * kefu (~kefu@114.92.96.253) has joined #ceph
[14:10] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) has joined #ceph
[14:17] * KungFuHamster1 (~mrapple@9YSAAAORD.tor-irc.dnsbl.oftc.net) has joined #ceph
[14:18] * bniver (~bniver@nat-pool-bos-u.redhat.com) has joined #ceph
[14:20] <TMM> if a PG is incomplete, is there any way I can see what parts are missing on an ec pool? I'm using hammer
[14:20] * kefu (~kefu@114.92.96.253) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[14:22] <The_Ball> TMM, maybe ceph pg dump?
[14:22] * ade (~abradshaw@dslb-094-223-085-225.094.223.pools.vodafone-ip.de) has joined #ceph
[14:22] * rdas (~rdas@121.244.87.116) Quit (Quit: Leaving)
[14:22] <TMM> The_Ball, doesn't look like it :-/
[14:27] * Plord (~PKO@14.139.82.6) Quit (Quit: Textual IRC Client: www.textualapp.com)
[14:28] <TMM> as far as I can see I have all 10 parts on some osd or another
[14:28] * ade (~abradshaw@dslb-094-223-085-225.094.223.pools.vodafone-ip.de) Quit (Quit: Too sexy for his shirt)
[14:33] * toMeloos (~toMeloos@53568B3D.cm-6-7c.dynamic.ziggo.nl) has joined #ceph
[14:34] <willi> infernalis no problem test done
[14:34] <willi> 60gw data written
[14:34] <willi> type=0
[14:34] <willi> what now?
[14:34] <willi> 60gb written i mean
[14:34] <willi> exactly the same config
[14:35] <willi> like in jewel
[14:35] <willi> 18 data servers / 3 racks / 3 mons / 3 clients
[14:36] <willi> rack powered off in jewel = client disruption rbd ls not possible
[14:36] <willi> rack powered off in nfernalis = client runs after 20-30 seconds pasued
[14:36] <willi> paused
[14:36] <willi> ceph osd dump | grep -i rbd
[14:36] <willi> pool 0 'rbd' replicated size 3 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 4096 pgp_num 4096 last_change 444 flags hashpspool stripe_width 0
[14:37] <willi> http://pastebin.com/3LWefLHN
[14:37] <willi> root@ceph-mon-1:~# ceph -s
[14:37] <willi> cluster 29b35687-7bbd-41ea-bed2-df4cf00bae42
[14:37] <willi> health HEALTH_WARN
[14:37] <willi> clock skew detected on mon.ceph-mon-2
[14:37] <willi> Monitor clock skew detected
[14:37] <willi> monmap e1: 3 mons at {ceph-mon-1=10.250.250.5:6789/0,ceph-mon-2=10.250.250.6:6789/0,ceph-mon-3=10.250.250.7:6789/0}
[14:37] <willi> election epoch 10, quorum 0,1,2 ceph-mon-1,ceph-mon-2,ceph-mon-3
[14:37] <willi> osdmap e745: 90 osds: 90 up, 90 in
[14:37] <willi> flags sortbitwise
[14:37] <willi> pgmap v2257: 4096 pgs, 1 pools, 69008 MB data, 17322 objects
[14:37] <willi> 229 GB used, 83562 GB / 83792 GB avail
[14:37] <willi> 4096 active+clean
[14:37] <willi> client io 36284 B/s rd, 94670 kB/s wr, 1389 op/s
[14:38] * i_m (~ivan.miro@deibp9eh1--blueice4n0.emea.ibm.com) has joined #ceph
[14:39] <willi> here
[14:39] <willi> powered off in jewel
[14:39] <willi> https://paste.ee/p/BtUN2
[14:39] <willi> powered off in infernalis
[14:39] <willi> https://paste.ee/p/BCBtx
[14:40] <willi> jewel line 258 and to the end there we have powered on again rack 1
[14:43] <willi> anyone ?
[14:46] * allaok (~allaok@machine107.orange-labs.com) has left #ceph
[14:47] * KungFuHamster1 (~mrapple@9YSAAAORD.tor-irc.dnsbl.oftc.net) Quit ()
[14:47] * KeeperOfTheSoul (~PappI@exit1.torproxy.org) has joined #ceph
[14:48] * jcsp (~jspray@82-71-16-249.dsl.in-addr.zen.co.uk) has joined #ceph
[14:48] * kawa2014 (~kawa@89.184.114.246) Quit (Ping timeout: 480 seconds)
[14:50] * vimal (~vikumar@121.244.87.116) Quit (Quit: Leaving)
[15:00] * mhack (~mhack@nat-pool-bos-t.redhat.com) has joined #ceph
[15:00] * kawa2014 (~kawa@212.110.41.244) has joined #ceph
[15:05] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) Quit (Quit: neurodrone_)
[15:07] * brad_mssw (~brad@66.129.88.50) has joined #ceph
[15:09] <willi> can anyboday confirm this?
[15:09] <willi> #Choose a reasonable crush leaf type
[15:09] <willi> #0 for a 1-node cluster.
[15:09] <willi> #1 for a multi node cluster in a single rack
[15:09] <willi> #2 for a multi node, multi chassis cluster with multiple hosts in a chassis
[15:09] <willi> #3 for a multi node cluster with hosts across racks, etc.
[15:09] <willi> osd crush chooseleaf type = 1
[15:09] * squizzi (~squizzi@107.13.31.195) has joined #ceph
[15:14] * spgriffinjr (~spgriffin@66.46.246.206) has joined #ceph
[15:15] <TMM> oh, pg query actually shows some of it
[15:15] * rnowling (~rnowling@104-186-210-225.lightspeed.milwwi.sbcglobal.net) has joined #ceph
[15:16] <TMM> I have imported some parts of the placement groups that I thought were damaged into a different osd
[15:16] <TMM> those are now showing that the imported data has 0 objects in it though
[15:16] <TMM> even though it is 6GB of 'stuff'
[15:17] * shyu (~Frank@218.241.172.114) Quit (Ping timeout: 480 seconds)
[15:17] * KeeperOfTheSoul (~PappI@61TAAANZ3.tor-irc.dnsbl.oftc.net) Quit ()
[15:20] * EthanL (~lamberet@cce02cs4035-fa12-z.ams.hpecore.net) Quit (Ping timeout: 480 seconds)
[15:29] * EthanL (~lamberet@cce02cs4035-fa12-z.ams.hpecore.net) has joined #ceph
[15:33] * vbellur (~vijay@2601:18f:700:55b0:5e51:4fff:fee8:6a5c) Quit (Ping timeout: 480 seconds)
[15:33] <TMM> I don't understand why this pg is incomplete. It seems that 7 of my k+m of 7+3 parts are available
[15:35] <The_Ball> TMM, I'm learning. How do you import part of a PG?
[15:35] * scg (~zscg@146-115-134-246.c3-0.nwt-ubr1.sbo-nwt.ma.cable.rcn.com) has joined #ceph
[15:38] <TMM> normally you don't
[15:38] <TMM> :)
[15:38] <TMM> but you can use ceph-objectstore-tool
[15:38] <TMM> but you really should never need to use that
[15:38] <TMM> probably good to play around with a test cluster with it though
[15:40] * bara (~bara@nat-pool-brq-t.redhat.com) Quit (Remote host closed the connection)
[15:42] * bara (~bara@nat-pool-brq-t.redhat.com) has joined #ceph
[15:44] <TMM> I have 64.es 2,3,5,6,7,8,9 all with the same timestamps according to pg query
[15:44] <TMM> why won't it just rebuild the missing data?
[15:44] * alexxy (~alexxy@biod.pnpi.spb.ru) Quit (Quit: No Ping reply in 180 seconds.)
[15:45] * alexxy (~alexxy@biod.pnpi.spb.ru) has joined #ceph
[15:47] * vbellur (~vijay@nat-pool-bos-t.redhat.com) has joined #ceph
[15:49] * dgurtner (~dgurtner@178.197.227.242) Quit (Ping timeout: 480 seconds)
[15:51] * mattbenjamin (~mbenjamin@12.118.3.106) has joined #ceph
[15:51] * dgurtner (~dgurtner@178.197.233.38) has joined #ceph
[15:58] <sep> willi, what chooseleaf type did you use ?
[15:59] <willi> 3
[16:03] <sep> willi, paste 'ceph osd tree'
[16:04] <willi> https://paste.ee/p/Y78Du
[16:07] * kefu (~kefu@114.92.96.253) has joined #ceph
[16:07] * IvanJobs (~ivanjobs@103.50.11.146) Quit (Remote host closed the connection)
[16:08] * IvanJobs (~ivanjobs@103.50.11.146) has joined #ceph
[16:09] * xarses (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[16:12] * yanzheng (~zhyan@125.70.23.222) Quit (Quit: This computer has gone to sleep)
[16:14] * Hemanth (~hkumar_@121.244.87.117) Quit (Ping timeout: 480 seconds)
[16:16] * DanFoster (~Daniel@2a00:1ee0:3:1337:f86c:d268:ead0:cef8) has joined #ceph
[16:16] * IvanJobs (~ivanjobs@103.50.11.146) Quit (Ping timeout: 480 seconds)
[16:17] * sixofour1 (~Fapiko@192.42.116.16) has joined #ceph
[16:19] * rnowling (~rnowling@104-186-210-225.lightspeed.milwwi.sbcglobal.net) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[16:19] <sep> looks good to me. I am afraid i am not experienced enough with running jewel to have any more input. either hang around waiting for people to wake in their timezones. or write to the mailinglist. assuming everything else is identical it looks like you might have found a degradation between infernalis->jewel , in the not so often used usecase of having a degraded cluster, with nowhere to backfill
[16:20] * kawa2014 (~kawa@212.110.41.244) Quit (Ping timeout: 480 seconds)
[16:21] * dan__ (~Daniel@2a00:1ee0:3:1337:a1cf:53f2:472a:9931) Quit (Ping timeout: 480 seconds)
[16:24] * rotbeard (~redbeard@aftr-109-90-233-215.unity-media.net) Quit (Quit: Leaving)
[16:25] * EinstCrazy (~EinstCraz@58.247.119.250) Quit (Remote host closed the connection)
[16:29] * kawa2014 (~kawa@89.184.114.246) has joined #ceph
[16:30] * rnowling (~rnowling@104-186-210-225.lightspeed.milwwi.sbcglobal.net) has joined #ceph
[16:31] * dougf (~dougf@96-38-99-179.dhcp.jcsn.tn.charter.com) Quit (Ping timeout: 480 seconds)
[16:32] * dougf (~dougf@96-38-99-179.dhcp.jcsn.tn.charter.com) has joined #ceph
[16:36] * mykola (~Mikolaj@91.245.77.88) has joined #ceph
[16:37] * penguinRaider (~KiKo@103.6.219.219) Quit (Ping timeout: 480 seconds)
[16:38] * johnavp1989 (~jpetrini@8.39.115.8) has joined #ceph
[16:38] <- *johnavp1989* To prove that you are human, please enter the result of 8+3
[16:38] * evelu (~erwan@37.160.182.21) has joined #ceph
[16:47] * penguinRaider (~KiKo@103.6.219.219) has joined #ceph
[16:47] * sixofour1 (~Fapiko@5AEAAAD5U.tor-irc.dnsbl.oftc.net) Quit ()
[16:47] * joshd1 (~jdurgin@2602:30a:c089:2b0:395d:ff46:593f:965f) has joined #ceph
[16:49] * vata (~vata@207.96.182.162) has joined #ceph
[16:51] * micw (~micw@p50992bfa.dip0.t-ipconnect.de) has joined #ceph
[16:52] * InIMoeK (~InIMoeK@95.170.93.16) has joined #ceph
[16:52] * xarses (~xarses@64.124.158.100) has joined #ceph
[16:52] <InIMoeK> hi guys, I have a strange issue
[16:52] <InIMoeK> after restarting one of my OSD machines
[16:53] <InIMoeK> I'm unable to start the osd because the journal partition cannot be mounted
[16:53] * maybebuggy (~maybebugg@2a01:4f8:191:2350::2) Quit (Ping timeout: 480 seconds)
[16:53] <InIMoeK> it's a multi journal partition on one NVME drive
[16:54] <InIMoeK> 2016-07-18 16:43:21.052309 7f52d3eb58c0 -1 filestore(/var/lib/ceph/osd/ceph-20) mount failed to open journal /var/lib/ceph/osd/ceph-20/journal: (13) Permission denied
[16:54] <InIMoeK> 2016-07-18 16:43:21.062507 7f52d3eb58c0 -1 osd.20 0 OSD:init: unable to mount object store
[16:54] <InIMoeK> 2016-07-18 16:43:21.062526 7f52d3eb58c0 -1 ** ERROR: osd init failed: (13) Permission denied
[16:54] * evelu (~erwan@37.160.182.21) Quit (Ping timeout: 480 seconds)
[16:55] <InIMoeK> I've tried to chown the /dev/nvme0n1p+ to ceph:ceph
[16:55] <InIMoeK> still no luck
[16:59] * kefu (~kefu@114.92.96.253) Quit (Max SendQ exceeded)
[16:59] * kefu (~kefu@114.92.96.253) has joined #ceph
[16:59] * debian112 (~bcolbert@207.183.247.46) Quit (Quit: Leaving.)
[17:00] <etienneme> Are you sure /var/lib/ceph/osd/ceph-20/journal rights are OK ?
[17:01] <TMM> InIMoeK, are you on centos or fedora?
[17:01] <T1> hopefully it's a symblink to the partition
[17:01] <T1> but yes - that's where I'd look first
[17:06] * wushudoin (~wushudoin@2601:646:8281:cfd:2ab2:bdff:fe0b:a6ee) has joined #ceph
[17:06] * vbellur (~vijay@nat-pool-bos-t.redhat.com) Quit (Ping timeout: 480 seconds)
[17:08] <InIMoeK> I'm on ubuntu 16.04
[17:08] <InIMoeK> yes it's a simlink to a partition
[17:08] * kefu (~kefu@114.92.96.253) Quit (Max SendQ exceeded)
[17:09] <InIMoeK> root@osd07:/home/cephadmin# ls -lah /var/lib/ceph/osd/ceph-20/journal
[17:09] <InIMoeK> lrwxrwxrwx 1 ceph ceph 14 Jul 14 15:26 /var/lib/ceph/osd/ceph-20/journal -> /dev/nvme0n1p1
[17:09] * gregmark (~Adium@68.87.42.115) has joined #ceph
[17:09] * swami1 (~swami@49.38.0.153) Quit (Quit: Leaving.)
[17:10] * kefu (~kefu@114.92.96.253) has joined #ceph
[17:10] <InIMoeK> somehow the permissions are reverting aswel
[17:10] <InIMoeK> root@osd07:/home/cephadmin# ls -lah /dev/nvme0n1p1
[17:10] <InIMoeK> brw-rw---- 1 root disk 259, 6 Jul 18 17:06 /dev/nvme0n1p1
[17:11] * ira (~ira@nat-pool-bos-u.redhat.com) Quit (Remote host closed the connection)
[17:12] <topro> InIMoeK: gpt partition table on journal drive?
[17:12] <InIMoeK> yes
[17:13] <InIMoeK> Command (m for help): p
[17:13] <InIMoeK> Disk /dev/nvme0n1: 372.6 GiB, 400088457216 bytes, 781422768 sectors
[17:13] <InIMoeK> Units: sectors of 1 * 512 = 512 bytes
[17:13] <InIMoeK> Sector size (logical/physical): 512 bytes / 512 bytes
[17:13] <InIMoeK> I/O size (minimum/optimal): 512 bytes / 512 bytes
[17:13] <InIMoeK> Disklabel type: gpt
[17:14] <topro> have you set the partitions GUID to "CEPH JOURNAL" (see https://en.wikipedia.org/wiki/GUID_Partition_Table, search for ceph) that will help ceph to set correct permissions using udev (IIRC)
[17:15] <InIMoeK> I'll check
[17:17] * rakeshgm (~rakesh@121.244.87.117) Quit (Quit: Leaving)
[17:17] * dgurtner (~dgurtner@178.197.233.38) Quit (Read error: No route to host)
[17:18] <InIMoeK> but I have 4 partitions on the nvme drive
[17:19] <InIMoeK> oh wait
[17:19] <InIMoeK> nvrmind
[17:19] * salwasser (~Adium@a72-246-0-10.deploy.akamaitechnologies.com) has joined #ceph
[17:19] * davidzlap (~Adium@cpe-172-91-154-245.socal.res.rr.com) has joined #ceph
[17:19] * shaunm (~shaunm@cpe-192-180-17-174.kya.res.rr.com) has joined #ceph
[17:20] <salwasser> Good morning, I have a question about a monmap that is resolutely refusing to compact. Would this or ceph-users be a better venue?
[17:21] * vbellur (~vijay@nat-pool-bos-u.redhat.com) has joined #ceph
[17:22] * dnunez (~dnunez@130.64.25.56) has joined #ceph
[17:23] <salwasser> sorry, monitor store, not monmap
[17:26] * EinstCrazy (~EinstCraz@58.247.119.250) has joined #ceph
[17:26] * borei (~dan@216.13.217.230) has joined #ceph
[17:30] * Jeffrey4l_ (~Jeffrey@119.251.244.121) Quit (Ping timeout: 480 seconds)
[17:32] <InIMoeK> topro thanks! I think this will solve my issue
[17:34] * EinstCrazy (~EinstCraz@58.247.119.250) Quit (Ping timeout: 480 seconds)
[17:35] <InIMoeK> I thought ceph deploy would take care of this
[17:37] * sudocat (~dibarra@192.185.1.20) has joined #ceph
[17:37] <topro> I'm not using ceph deploy, doing manual setup, so I cannot tell. but maybe you did the setup with too old a version of ceph-deploy as that guid stuff is quite new AFIAK
[17:38] <InIMoeK> I see
[17:38] <InIMoeK> it was setup this month
[17:38] <InIMoeK> from the normal repo's
[17:38] * blizzow (~jburns@50.243.148.102) has joined #ceph
[17:38] <InIMoeK> so in this case the disk contains 5 partitions
[17:39] <InIMoeK> 4 journals and one readcache parition
[17:39] <InIMoeK> do you have to set the GUID for the disk itself only
[17:39] <InIMoeK> or all the partitions
[17:39] * mykola (~Mikolaj@91.245.77.88) Quit (Read error: Connection reset by peer)
[17:39] <InIMoeK> except for the readcache one
[17:39] <topro> set GUID per partition
[17:39] <InIMoeK> thank you
[17:40] * mykola (~Mikolaj@91.245.77.88) has joined #ceph
[17:40] <topro> I did it using gdisk setting partition type of my journal partitions to f802 which corresponds to the right GUID. and magically ceph sets permissions to ceph:ceph instantaneously
[17:41] <topro> what i did additionally afterwards was a partprobe, but not sure if really needed
[17:46] * linuxkidd (~linuxkidd@ip70-189-207-54.lv.lv.cox.net) has joined #ceph
[17:47] * TMM (~hp@185.5.121.201) Quit (Quit: Ex-Chat)
[17:47] * vikhyat (~vumrao@121.244.87.116) Quit (Quit: Leaving)
[17:48] <InIMoeK> thanks topro
[17:48] <InIMoeK> I'll start changing thant
[17:48] <InIMoeK> that*
[17:48] * bara (~bara@nat-pool-brq-t.redhat.com) Quit (Quit: Bye guys! (??????????????????? ?????????)
[17:50] * Hemanth (~hkumar_@103.228.221.160) has joined #ceph
[17:51] * kawa2014 (~kawa@89.184.114.246) Quit (Quit: Leaving)
[17:55] * sudocat (~dibarra@192.185.1.20) Quit (Ping timeout: 480 seconds)
[17:56] * InIMoeK (~InIMoeK@95.170.93.16) Quit (Ping timeout: 480 seconds)
[18:00] * sudocat (~dibarra@192.185.1.20) has joined #ceph
[18:00] * mashwo00 (~textual@51.179.162.234) Quit (Quit: Textual IRC Client: www.textualapp.com)
[18:02] * MatthewH12 (~kalmisto@194.187.249.135) has joined #ceph
[18:03] * b0e (~aledermue@213.95.25.82) Quit (Quit: Leaving.)
[18:03] * debian112 (~bcolbert@2600:1010:b003:8172:863a:4bff:fe9b:d1ba) has joined #ceph
[18:05] * sudocat (~dibarra@192.185.1.20) Quit ()
[18:05] * sudocat (~dibarra@192.185.1.20) has joined #ceph
[18:12] * dmick (~dmick@206.169.83.146) has left #ceph
[18:15] * debian112 (~bcolbert@2600:1010:b003:8172:863a:4bff:fe9b:d1ba) has left #ceph
[18:16] * debian112 (~bcolbert@2600:1010:b003:8172:863a:4bff:fe9b:d1ba) has joined #ceph
[18:16] * allaok1 (~allaok@machine107.orange-labs.com) has joined #ceph
[18:16] * gwinger (~gwinger@2a02:8108:42c0:a68:9030:18e9:60c0:bd5a) has joined #ceph
[18:17] * derjohn_mob (~aj@2001:6f8:1337:0:b56f:1c5c:41a8:95c1) Quit (Ping timeout: 480 seconds)
[18:18] * gwinger (~gwinger@2a02:8108:42c0:a68:9030:18e9:60c0:bd5a) Quit ()
[18:18] * gwinger (~gwinger@2a02:8108:42c0:a68:9030:18e9:60c0:bd5a) has joined #ceph
[18:19] * gwinger (~gwinger@2a02:8108:42c0:a68:9030:18e9:60c0:bd5a) Quit ()
[18:19] * gwinger (~gwinger@ip5f5be40b.dynamic.kabel-deutschland.de) has joined #ceph
[18:22] * sudocat (~dibarra@192.185.1.20) Quit (Quit: Leaving.)
[18:22] * sudocat (~dibarra@192.185.1.20) has joined #ceph
[18:23] * MentalRay (~MentalRay@office-mtl1-nat-146-218-70-69.gtcomm.net) has joined #ceph
[18:26] * swami1 (~swami@27.7.162.30) has joined #ceph
[18:26] * gregsfortytwo1 (~gregsfort@transit-86-181-132-209.redhat.com) Quit (Quit: bye!)
[18:28] * newbie (~kvirc@host217-114-156-249.pppoe.mark-itt.net) has joined #ceph
[18:29] * Skaag (~lunix@65.200.54.234) has joined #ceph
[18:29] * debian112 (~bcolbert@2600:1010:b003:8172:863a:4bff:fe9b:d1ba) Quit (Ping timeout: 480 seconds)
[18:31] * MatthewH12 (~kalmisto@26XAAAER2.tor-irc.dnsbl.oftc.net) Quit ()
[18:34] * karnan (~karnan@121.244.87.117) Quit (Remote host closed the connection)
[18:35] * ngoswami_ (~ngoswami@1.39.87.139) has joined #ceph
[18:39] * vbellur (~vijay@nat-pool-bos-u.redhat.com) Quit (Ping timeout: 480 seconds)
[18:41] * ngoswami (~ngoswami@1.39.87.139) Quit (Ping timeout: 480 seconds)
[18:42] * debian112 (~bcolbert@64.235.154.81) has joined #ceph
[18:42] * arcimboldo (~antonio@dhcp-y11-zi-s3it-130-60-34-054.uzh.ch) Quit (Quit: Ex-Chat)
[18:43] * hybrid512 (~walid@195.200.189.206) Quit (Remote host closed the connection)
[18:43] * i_m (~ivan.miro@deibp9eh1--blueice4n0.emea.ibm.com) Quit (Ping timeout: 480 seconds)
[18:45] * Kizzi (~shishi@Relay-J.tor-exit.network) has joined #ceph
[18:47] * vasu (~vasu@c-73-231-60-138.hsd1.ca.comcast.net) has joined #ceph
[18:49] * rraja (~rraja@121.244.87.117) Quit (Quit: Leaving)
[18:52] * vbellur (~vijay@nat-pool-bos-t.redhat.com) has joined #ceph
[18:55] * joshd1 (~jdurgin@2602:30a:c089:2b0:395d:ff46:593f:965f) Quit (Quit: Leaving.)
[18:56] * fdmanana (~fdmanana@2001:8a0:6e0c:6601:7981:6bc4:cb04:3446) Quit (Ping timeout: 480 seconds)
[18:57] * squizzi_ (~squizzi@107.13.31.195) has joined #ceph
[18:57] * squizzi (~squizzi@107.13.31.195) Quit (Ping timeout: 480 seconds)
[18:57] * mhackett (~mhack@nat-pool-bos-u.redhat.com) has joined #ceph
[18:58] * gregsfortytwo (~gregsfort@transit-86-181-132-209.redhat.com) has joined #ceph
[19:00] * jermudgeon (~jhaustin@31.207.56.59) has joined #ceph
[19:00] * kefu (~kefu@114.92.96.253) Quit (Max SendQ exceeded)
[19:00] * kefu (~kefu@114.92.96.253) has joined #ceph
[19:03] * mhack (~mhack@nat-pool-bos-t.redhat.com) Quit (Ping timeout: 480 seconds)
[19:04] * vbellur (~vijay@nat-pool-bos-t.redhat.com) Quit (Ping timeout: 480 seconds)
[19:05] * allaok (~allaok@machine107.orange-labs.com) has joined #ceph
[19:09] * allaok1 (~allaok@machine107.orange-labs.com) Quit (Ping timeout: 480 seconds)
[19:15] * Kizzi (~shishi@61TAAAN83.tor-irc.dnsbl.oftc.net) Quit ()
[19:17] * vbellur (~vijay@nat-pool-bos-u.redhat.com) has joined #ceph
[19:28] * EinstCrazy (~EinstCraz@58.247.119.250) has joined #ceph
[19:29] * willi (~willi@p200300774E3477FC642DA7132ED418F6.dip0.t-ipconnect.de) Quit ()
[19:30] * kefu is now known as kefu|afk
[19:35] * kefu|afk (~kefu@114.92.96.253) Quit (Max SendQ exceeded)
[19:35] * MACscr (~Adium@c-73-9-230-5.hsd1.il.comcast.net) has joined #ceph
[19:36] * EinstCrazy (~EinstCraz@58.247.119.250) Quit (Ping timeout: 480 seconds)
[19:36] * stiopa (~stiopa@cpc73832-dals21-2-0-cust453.20-2.cable.virginm.net) has joined #ceph
[19:38] <MACscr> im getting HEALTH_WARN mon.1 low disk space, but none of my disks are even close to full. the closest is a partition that is at 68%, but i wouldnt really call that a warning level yet
[19:39] * branto (~branto@ip-78-102-208-181.net.upcbroadband.cz) Quit (Quit: Leaving.)
[19:44] * bniver (~bniver@nat-pool-bos-u.redhat.com) Quit (Remote host closed the connection)
[19:44] * mhack (~mhack@nat-pool-bos-t.redhat.com) has joined #ceph
[19:45] * treenerd_ (~gsulzberg@cpe90-146-148-47.liwest.at) has joined #ceph
[19:45] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[19:48] * davidzlap (~Adium@cpe-172-91-154-245.socal.res.rr.com) Quit (Quit: Leaving.)
[19:49] * mhackett (~mhack@nat-pool-bos-u.redhat.com) Quit (Ping timeout: 480 seconds)
[19:50] <TheSov> whats the size of /var
[19:50] <TheSov> MACscr,
[19:51] * ngoswami_ (~ngoswami@1.39.87.139) Quit (Quit: This computer has gone to sleep)
[19:52] <MACscr> TheSov: http://paste.debian.net/plain/782835
[19:53] <TheSov> ... /dev/sdd1 7.9G 4.9G 2.6G 66% /
[19:53] <TheSov> now disk space
[19:53] <TheSov> low*
[19:53] <TheSov> 2.6 gigs left
[19:53] * ngoswami (~ngoswami@1.39.87.139) has joined #ceph
[19:54] <MACscr> and? thats plenty
[19:54] <MACscr> the os disks are only 8gb themselves. lol
[19:54] <Anticimex> so i have 4 ssd nodes. 1 is out, cluster is noout.
[19:54] <TheSov> i know
[19:54] <TheSov> but thats below threshold for monitors
[19:55] <TheSov> remember the monitors keep a map of all data
[19:55] <MACscr> TheSov: determined by what?
[19:55] <TheSov> good question
[19:55] <Anticimex> as soon as i start bringing osds on the down node back again, even a single one, backfill and recovery starts, and requests starts to back up. there seems to be zero movement
[19:55] <Anticimex> this for a pg where the "coming up"-OSD is the 3rd and last in the acting set
[19:55] <TheSov> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-January/036751.html
[19:55] <Anticimex> infernalis. and weird
[19:55] <TheSov> there MACscr
[19:55] * vata (~vata@207.96.182.162) Quit (Quit: Leaving.)
[19:56] <Anticimex> gonna test with osd backfills etc now
[19:56] <TheSov> Anticimex, whats a node?
[19:56] <TheSov> do you mean osds?
[19:56] <Anticimex> sorry, a server
[19:56] <Anticimex> with 24 osds on
[19:56] <TheSov> ok
[19:56] <Anticimex> one of them has all osds down, to use better terminology
[19:57] <TheSov> ok
[19:57] <TheSov> do you have default crush rules?
[19:57] <Anticimex> i've been bumbing them up and down all day and yesterday, haven't seen this before
[19:57] <Anticimex> not really, but quite standard for the osds on these servers. 3 replicas
[19:58] <Anticimex> balancing etc has worked fine before, the osds on this server has ~25% of the data
[19:58] <TheSov> so you up an osd
[19:58] <TheSov> and then everything locks up?
[19:58] <Anticimex> for some definition of 'everything'
[19:58] <Anticimex> i suppose ops against a specific set of pgs only
[19:58] <TheSov> what are your pg recovery limits?
[19:58] <Anticimex> though i'm not sure that's what i see
[19:58] <T1> what does your crush rules look like?
[19:59] <Anticimex> but there's a single pg that's backfilling
[19:59] * TMM (~hp@178-84-46-106.dynamic.upc.nl) has joined #ceph
[19:59] <MACscr> TheSov: says the default is 30% for warn, which i havent hit, so still not sure why it would be warning right now
[19:59] <Anticimex> and the osd reports increasing amount of blocked requests
[19:59] <Anticimex> which params do you want to see?
[19:59] <T1> I can't remember the exact details, but it's possible to have all OSDs for a single PG placed within a single node
[19:59] <TheSov> MACscr, ? 8 * .3 = 2.4
[20:00] <T1> .. which is bad
[20:00] <Anticimex> T1: they're not, they are spaced out on all pgs that i checked here
[20:00] <Anticimex> 1 server each, crush appears to do its job
[20:00] <T1> Anticimex: ok, so the PG that's requested can be served by OSDs that are not down/out?
[20:00] <Anticimex> yeah, this particular pg that goes backfilling is the 3rd in the acting set
[20:00] <TheSov> osd max backfills
[20:00] <TheSov> and osd recovery threads
[20:01] <Anticimex> i.e the primary osd and osd nr 2 is on a different host entirely
[20:01] <T1> ok
[20:01] <TheSov> Anticimex, give me the output of your ceph osd tree
[20:01] <T1> is it just this one PG or is it IO in general thats blocked?
[20:01] <Anticimex> the primary osd goes into blocked requests
[20:01] <Anticimex> i guess that's the donor OSD when backfilling
[20:02] <Anticimex> and no, only within this pg
[20:02] <Anticimex> TheSov: sure, give me a few moments
[20:02] <T1> hm, strange
[20:03] <Anticimex> i haven't done all the investigation i can yet myself, but it's weird enough for me i thought i'd ask
[20:03] <Anticimex> TheSov: first those two params:
[20:03] <Anticimex> "osd_max_backfills": "1",
[20:04] <Anticimex> "osd_recovery_threads": "1",
[20:04] <Anticimex> no science at all behind those, i think they're defaults?
[20:04] <TheSov> yes
[20:04] <TheSov> i thought you had them set really high or something
[20:05] <Anticimex> like i said i've been doing this same process many times, like 10:ish, past 48h
[20:06] <Anticimex> only difference i can really think between today and yesterday, where iwas doing many is that the amount of data to backfill is larger now since it's been a work day
[20:06] <Anticimex> but i wouldn't expect the primary osd in the pg to lock up *shrug*
[20:08] <Anticimex> TheSov: http://martin.millnert.se/files/osd.tree.txt
[20:08] <Anticimex> it's the 4 ceph-prod-dc1ssd* hosts that are involved
[20:11] <MACscr> TheSov: how do i inject the new warning levels on a live monitor? I already adjusted the conf
[20:11] <TheSov> ok Anticimex when you turn it ok
[20:11] <TheSov> what exactly happens
[20:11] <TheSov> let me look
[20:12] <Anticimex> TheSov: first piece of data is the log entry on the pg that's actively backfilling/recovering, from the primary osd in the pg
[20:12] <Anticimex> 2016-07-18 19:10:46.599481 7f8b45bd3700 0 log_channel(cluster) log [WRN] : 6 slow requests, 6 included below; oldest blocked for > 30.849550 secs
[20:12] <Anticimex> 2016-07-18 19:10:46.599496 7f8b45bd3700 0 log_channel(cluster) log [WRN] : slow request 30.848194 seconds old, received at 2016-07-18 19:10:15.751177: osd_op(client.16145255.0:1992566 rbd_data.f4519327aaecc8.00000000000002c1 [write 2928640~4096] 20.56cce1 ack+ondisk+write+known_if_redirected e31937) currently waiting for degraded object
[20:13] <Anticimex> this is from when i turned it on 1h ago. it's live cluster so don't want to block io too long ;)
[20:14] <TheSov> hmmmm
[20:14] <TheSov> whats ceph -s say
[20:15] * wjw-freebsd (~wjw@smtp.digiware.nl) has joined #ceph
[20:15] <Anticimex> right now it's only saying:
[20:15] <Anticimex> 5756 active+clean
[20:15] <Anticimex> 3170 active+undersized+degraded
[20:15] <Anticimex> 2 active+clean+scrubbing
[20:15] * vbellur (~vijay@nat-pool-bos-u.redhat.com) Quit (Ping timeout: 480 seconds)
[20:15] * haplo37 (~haplo37@199.91.185.156) has joined #ceph
[20:15] <Anticimex> where all those 3170 undersized+degraded pgs are due to the ssd04 host
[20:15] * karnan (~karnan@106.51.130.90) has joined #ceph
[20:15] <Anticimex> (the scrubbing is on some other rack of hosts)
[20:15] <Anticimex> i turned off deep scrub anyway
[20:16] <Anticimex> but i'm not running any osd from the ssd04 host atm
[20:18] <TheSov> well undersized could also mean your crush rules need more systems
[20:18] <Anticimex> i have to start one up but i want to collect some data
[20:18] <TheSov> when its off you are clear of all errors?
[20:18] <Anticimex> yeah, i was clear of errors just some hours ago
[20:18] <Anticimex> put noout, and downed all osds on this ssd04 host
[20:20] <Anticimex> i guess i should debug a bit from the osds that are member of this pg that is seemingly stalling io
[20:22] * shylesh__ (~shylesh@45.124.226.156) has joined #ceph
[20:26] * jermudgeon (~jhaustin@31.207.56.59) Quit (Quit: jermudgeon)
[20:27] * vbellur (~vijay@nat-pool-bos-t.redhat.com) has joined #ceph
[20:28] * fdmanana (~fdmanana@2001:8a0:6e0c:6601:e189:abfd:7eae:1796) has joined #ceph
[20:28] <TheSov> well my suggestion at this point, you wouldnt like lol
[20:29] <Anticimex> hmm
[20:29] * ngoswami (~ngoswami@1.39.87.139) Quit (Quit: Leaving)
[20:29] <TheSov> but ill tell you. delete those osd's from the cluster bring the unit up
[20:29] <Anticimex> from that log above
[20:29] <TheSov> and re add them
[20:29] <Anticimex> hehe, i can do that, but i want to understand what's wrong
[20:29] <Anticimex> i think there's a diff fix
[20:29] <Anticimex> so in that log above, it says "20.56cce1 ack+ondisk+write+known_if_redirected"
[20:29] <Anticimex> i suppose 20.56cce1 is supposed to be poolnum.pgnum
[20:30] <Anticimex> but... that's not a format i recognize for pgnum, don't have that many
[20:34] <MACscr> TheSov: any ideas how to change that warning level on the fly?
[20:35] <MACscr> do i just run ceph ???mon-data-avail-warn 20
[20:36] <MACscr> well with the double dash. irc client converts it for some reason
[20:43] * karnan (~karnan@106.51.130.90) Quit (Quit: Leaving)
[20:44] * karnan (~karnan@106.51.130.90) has joined #ceph
[20:45] * sudocat (~dibarra@192.185.1.20) Quit (Remote host closed the connection)
[20:45] * cathode (~cathode@50.232.215.114) has joined #ceph
[20:45] * fdmanana (~fdmanana@2001:8a0:6e0c:6601:e189:abfd:7eae:1796) Quit (Ping timeout: 480 seconds)
[20:46] * sudocat (~dibarra@192.185.1.20) has joined #ceph
[20:47] <Anticimex> TheSov: this seems similar, http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-December/006664.html
[20:49] * KungFuHamster (~Esvandiar@93.115.95.201) has joined #ceph
[20:51] * sudocat (~dibarra@192.185.1.20) Quit ()
[20:51] * sudocat (~dibarra@192.185.1.20) has joined #ceph
[20:51] <T1> what version did you say you were running?
[20:52] <T1> hammer, infernalis or jewel?
[20:57] * sudocat (~dibarra@192.185.1.20) Quit (Remote host closed the connection)
[20:58] <Anticimex> infernalis
[21:00] * cathode (~cathode@50.232.215.114) Quit (Quit: Leaving)
[21:02] <T1> I'd look for a possible bugfix then..
[21:02] <T1> that mailinglist entry is ~1 month after infernalis came out
[21:03] * bene2 (~bene@nat-pool-bos-t.redhat.com) Quit (Quit: Konversation terminated!)
[21:03] * swami1 (~swami@27.7.162.30) Quit (Quit: Leaving.)
[21:04] <Anticimex> right, the problem is sort of core ceph architecture though:s
[21:05] <Anticimex> but i'll try to figure out if that's the problem, still need to better understand what's happening when i turn an osd on (i.e. if there's lots of throughput of data or whatever)
[21:05] <Anticimex> it's not exactly identical situation
[21:05] * Hemanth (~hkumar_@103.228.221.160) Quit (Quit: Leaving)
[21:06] * rendar (~I@host1-139-dynamic.49-82-r.retail.telecomitalia.it) Quit (Ping timeout: 480 seconds)
[21:08] <Anticimex> the reported blockage times are due to reading pgs from disk in that thread, i think, which i doubt is the issue here :s i had many minutes of block, way more than the time it takes to read from the ssds in question with the data they have
[21:13] * sudocat (~dibarra@192.185.1.20) has joined #ceph
[21:13] * cyphase_eviltwin (~cyphase@c-50-148-131-137.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[21:15] * cyphase (~cyphase@000134f2.user.oftc.net) has joined #ceph
[21:15] <Anticimex> hmm, i have a clue
[21:16] <Anticimex> so, i attempted to bring up 1 osd again.
[21:17] <Anticimex> it logs "slow request .... currently waiting for missing object"
[21:17] <Anticimex> so as soon as it completes its initializating routine, it goes directly from downtime to active primary for all its pgs
[21:17] <Anticimex> though it's behind, and don't have those objects yet
[21:18] <Anticimex> and meanwhile ceph -s said: "2 pgs recovering, 38 pgs recovery_wait, 1 pgs backfill"
[21:19] * KungFuHamster (~Esvandiar@9YSAAAO4D.tor-irc.dnsbl.oftc.net) Quit ()
[21:20] * slowriot (~kiasyn@tor-exit.talyn.se) has joined #ceph
[21:27] <T1> Animazing: that matches the ML..
[21:28] <T1> " ... The problem is that when
[21:28] <T1> the OSD is "in", the clients also see that and start sending requests
[21:28] <T1> to the OSDs before it has had a chance to actually get its bearings
[21:28] <T1> and is able to even service the requests. ... "
[21:29] <Anticimex> right
[21:29] <Anticimex> thouigh in ML one of the posters' reason for being slow was just bottlenecked IO. i don't see that going on here. rather just throttled recovery
[21:29] <Anticimex> tricky tradeoff that :)
[21:29] <Anticimex> i have slow recovery for other reason, tbd
[21:30] <Anticimex> RTFM:ing some related osd parameters now
[21:32] * karnan (~karnan@106.51.130.90) Quit (Quit: Leaving)
[21:32] * rendar (~I@host1-139-dynamic.49-82-r.retail.telecomitalia.it) has joined #ceph
[21:33] * karnan (~karnan@106.51.130.90) has joined #ceph
[21:40] * squizzi (~squizzi@107.13.31.195) has joined #ceph
[21:45] * karnan (~karnan@106.51.130.90) Quit (Quit: Leaving)
[21:45] * EthanL (~lamberet@cce02cs4035-fa12-z.ams.hpecore.net) Quit (Ping timeout: 480 seconds)
[21:47] * fdmanana (~fdmanana@2001:8a0:6e0c:6601:e189:abfd:7eae:1796) has joined #ceph
[21:47] * vata (~vata@207.96.182.162) has joined #ceph
[21:49] * slowriot (~kiasyn@26XAAAEYH.tor-irc.dnsbl.oftc.net) Quit ()
[21:54] * bniver (~bniver@pool-173-48-58-27.bstnma.fios.verizon.net) has joined #ceph
[21:55] * EthanL (~lamberet@cce02cs4035-fa12-z.ams.hpecore.net) has joined #ceph
[21:59] * ircolle (~Adium@166.170.20.108) has joined #ceph
[22:00] * fdmanana (~fdmanana@2001:8a0:6e0c:6601:e189:abfd:7eae:1796) Quit (Ping timeout: 480 seconds)
[22:04] * ska (~skatinolo@cpe-173-174-111-177.austin.res.rr.com) has joined #ceph
[22:04] <ska> For remote file sharing, what is the most common setup that uses Ceph as a corporate solution?
[22:05] * salwasser (~Adium@a72-246-0-10.deploy.akamaitechnologies.com) Quit (Ping timeout: 480 seconds)
[22:05] <ska> I have 5 or so offices that need some sort of unified view of all the data.
[22:06] <Anticimex> rados gateway (S3/Swift) with some application on top? perhaps owncloud?
[22:07] * lobstar (~verbalins@edwardsnowden1.torservers.net) has joined #ceph
[22:08] * GeoTracer (~Geoffrey@41.77.153.99) Quit (Ping timeout: 480 seconds)
[22:08] * GeoTracer (~Geoffrey@41.77.153.99) has joined #ceph
[22:18] <TheSov> Anticimex, what happens if you re in 1 osd at a time
[22:19] <Anticimex> that's what i'm doing
[22:19] <TheSov> and is it working better than all in at once?
[22:20] <TheSov> for future reference no more noout
[22:20] * garphy is now known as garphy`aw
[22:20] * mykola (~Mikolaj@91.245.77.88) Quit (Quit: away)
[22:20] <Anticimex> well, not sure how to know for sure. when i 'in' all 24 osds the aggregated number of blocked ops is like 24x, so customer impact likely larger
[22:21] <ska> I guess you can use RGW/S3 like a samba share? Are there windows clients that can write to a master zone only, but read from local?
[22:21] <Anticimex> well, i'd be most happy if ceph devs actually fixed the state diagram/flow for 'in':ing an osd as per that ML above
[22:21] * brad_mssw (~brad@66.129.88.50) Quit (Quit: Leaving)
[22:22] * bene2 (~bene@nat-pool-bos-t.redhat.com) has joined #ceph
[22:28] * dyasny (~dyasny@cable-192.222.152.136.electronicbox.net) Quit (Ping timeout: 480 seconds)
[22:28] * rnowling (~rnowling@104-186-210-225.lightspeed.milwwi.sbcglobal.net) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[22:30] * karnan (~karnan@106.51.130.90) has joined #ceph
[22:30] * derjohn_mob (~aj@x590e62fa.dyn.telefonica.de) has joined #ceph
[22:31] <ska> Is there some way to instruct Ceph writes to non-master zones to be written to master?
[22:32] * dougf (~dougf@96-38-99-179.dhcp.jcsn.tn.charter.com) Quit (Ping timeout: 480 seconds)
[22:33] * squizzi_ (~squizzi@107.13.31.195) Quit (Quit: bye)
[22:33] * dougf (~dougf@96-38-99-179.dhcp.jcsn.tn.charter.com) has joined #ceph
[22:37] * lobstar (~verbalins@26XAAAEZ6.tor-irc.dnsbl.oftc.net) Quit ()
[22:38] * Shnaw (~elt@178-17-170-253.static.host) has joined #ceph
[22:38] * shylesh__ (~shylesh@45.124.226.156) Quit (Remote host closed the connection)
[22:39] * dyasny (~dyasny@cable-192.222.152.136.electronicbox.net) has joined #ceph
[22:39] * rnowling (~rnowling@104-186-210-225.lightspeed.milwwi.sbcglobal.net) has joined #ceph
[22:40] * EthanL (~lamberet@cce02cs4035-fa12-z.ams.hpecore.net) Quit (Read error: Connection reset by peer)
[22:43] * ircolle (~Adium@166.170.20.108) Quit (Quit: Leaving.)
[22:46] * dgurtner (~dgurtner@5.32.72.140) has joined #ceph
[22:51] * analbeard (~shw@support.memset.com) has joined #ceph
[22:52] * InIMoeK (~InIMoeK@105-183-045-062.dynamic.caiway.nl) has joined #ceph
[22:52] <InIMoeK> Hey topro
[22:52] <InIMoeK> it's working
[22:52] <InIMoeK> thanks again!
[22:53] <InIMoeK> I ended up doing :
[22:53] <InIMoeK> sudo sgdisk -t 1:45B0969E-9B03-4F30-B4C6-B4B80CEFF106 /dev/nvme0n1
[22:53] <InIMoeK> and then change the number in front of the GUID to set partition
[22:53] <InIMoeK> source http://tracker.ceph.com/issues/13833
[22:57] * vbellur (~vijay@nat-pool-bos-t.redhat.com) Quit (Quit: Leaving.)
[23:01] * newbie (~kvirc@host217-114-156-249.pppoe.mark-itt.net) Quit (Ping timeout: 480 seconds)
[23:02] * treenerd_ (~gsulzberg@cpe90-146-148-47.liwest.at) Quit (Quit: treenerd_)
[23:02] * GeoTracer (~Geoffrey@41.77.153.99) Quit (Ping timeout: 480 seconds)
[23:03] * GeoTracer (~Geoffrey@41.77.153.99) has joined #ceph
[23:06] * dan__ (~Daniel@2a00:1ee0:3:1337:2879:3fee:1f90:5474) has joined #ceph
[23:07] * Shnaw (~elt@9YSAAAO62.tor-irc.dnsbl.oftc.net) Quit ()
[23:08] * csharp (~DougalJac@watchme.tor-exit.network) has joined #ceph
[23:09] * DanFoster (~Daniel@2a00:1ee0:3:1337:f86c:d268:ead0:cef8) Quit (Ping timeout: 480 seconds)
[23:10] * cathode (~cathode@50.232.215.114) has joined #ceph
[23:17] * haplo37 (~haplo37@199.91.185.156) Quit (Remote host closed the connection)
[23:18] * dgurtner (~dgurtner@5.32.72.140) Quit (Ping timeout: 480 seconds)
[23:22] * dnunez (~dnunez@130.64.25.56) Quit (Quit: Leaving)
[23:23] * kwork (~quassel@bnc.ee) has joined #ceph
[23:31] * EinstCrazy (~EinstCraz@58.247.119.250) has joined #ceph
[23:33] * _28_ria (~kvirc@opfr028.ru) has joined #ceph
[23:36] * toMeloos (~toMeloos@53568B3D.cm-6-7c.dynamic.ziggo.nl) Quit (Quit: Ik ga weg)
[23:37] * csharp (~DougalJac@9YSAAAO7X.tor-irc.dnsbl.oftc.net) Quit ()
[23:39] * EinstCrazy (~EinstCraz@58.247.119.250) Quit (Ping timeout: 480 seconds)
[23:40] * reed (~reed@184-23-0-196.dsl.static.fusionbroadband.com) has joined #ceph
[23:43] * mattbenjamin (~mbenjamin@12.118.3.106) Quit (Ping timeout: 480 seconds)
[23:46] * salwasser1 (~Adium@2601:197:101:5cc1:85fb:8652:f30d:b653) has joined #ceph
[23:50] * salwasser1 (~Adium@2601:197:101:5cc1:85fb:8652:f30d:b653) Quit ()
[23:51] * shaunm (~shaunm@cpe-192-180-17-174.kya.res.rr.com) Quit (Ping timeout: 480 seconds)
[23:52] * omar_m (~omar_m@G68-121-13-92.sbcis.sbc.com) has joined #ceph
[23:54] * omar_m (~omar_m@G68-121-13-92.sbcis.sbc.com) has left #ceph
[23:57] * theTrav (~theTrav@203.35.9.142) has joined #ceph

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.