#ceph IRC Log

Index

IRC Log for 2016-04-07

Timestamps are in GMT/BST.

[0:00] * espeer (~quassel@phobos.isoho.st) Quit (Read error: Connection reset by peer)
[0:02] * espeer (~quassel@phobos.isoho.st) has joined #ceph
[0:02] * Skaag (~lunix@65.200.54.234) Quit (Quit: Leaving.)
[0:02] * thomnico (~thomnico@132-110.80-90.static-ip.oleane.fr) has joined #ceph
[0:03] * Skaag (~lunix@65.200.54.234) has joined #ceph
[0:04] * Discovery (~Discovery@178.239.49.70) Quit (Read error: Connection reset by peer)
[0:06] * fsimonce (~simon@host201-70-dynamic.26-79-r.retail.telecomitalia.it) Quit (Quit: Coyote finally caught me)
[0:07] * gregmark (~Adium@68.87.42.115) Quit (Quit: Leaving.)
[0:08] * ira (~ira@nat-pool-bos-t.redhat.com) Quit (Quit: Leaving)
[0:09] * gopher_49 (~gopher_49@host2.drexchem.com) has joined #ceph
[0:09] * gopher_49 (~gopher_49@host2.drexchem.com) Quit ()
[0:09] * vbellur1 (~vijay@2601:18f:702:3300::3b14) has joined #ceph
[0:10] * vbellur (~vijay@71.234.224.255) Quit (Ping timeout: 480 seconds)
[0:17] * mlovell (~mlovell@69-195-66-94.unifiedlayer.com) has joined #ceph
[0:18] * vbellur (~vijay@71.234.224.255) has joined #ceph
[0:19] <mlovell> is there anyone around that could give me some pointers on troubleshooting a ceph cluster? i have a cluster with a bunch of rbd clients and there is a lot of slow/blocked io requests and i can't figure out why its happening.
[0:19] * vbellur1 (~vijay@2601:18f:702:3300::3b14) Quit (Ping timeout: 480 seconds)
[0:20] * linjan_ (~linjan@176.195.217.66) has joined #ceph
[0:20] * Kizzi (~Wizeon@06SAAA09K.tor-irc.dnsbl.oftc.net) Quit ()
[0:22] * Skaag (~lunix@65.200.54.234) Quit (Quit: Leaving.)
[0:23] * haplo37 (~haplo37@199.91.185.156) Quit (Remote host closed the connection)
[0:26] * Skaag (~lunix@65.200.54.234) has joined #ceph
[0:27] * Skaag (~lunix@65.200.54.234) Quit ()
[0:27] * thomnico (~thomnico@132-110.80-90.static-ip.oleane.fr) Quit (Quit: Ex-Chat)
[0:27] * linjan__ (~linjan@176.193.197.8) Quit (Ping timeout: 480 seconds)
[0:27] * rendar (~I@host27-192-dynamic.181-80-r.retail.telecomitalia.it) Quit (Quit: std::lower_bound + std::less_equal *works* with a vector without duplicates!)
[0:28] * mhack (~mhack@66-168-117-78.dhcp.oxfr.ma.charter.com) has joined #ceph
[0:31] * bniver (~bniver@pool-173-48-58-27.bstnma.fios.verizon.net) has joined #ceph
[0:34] * fdmanana (~fdmanana@2001:8a0:6e0c:6601:60fc:c21d:c1c6:96e4) Quit (Ping timeout: 480 seconds)
[0:35] * BrianA1 (~BrianA@fw-rw.shutterfly.com) Quit (Read error: Connection reset by peer)
[0:36] * Skaag (~lunix@65.200.54.234) has joined #ceph
[0:37] * dgurtner_ (~dgurtner@217.149.140.193) has joined #ceph
[0:38] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[0:39] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[0:39] * dgurtner (~dgurtner@217.149.140.193) Quit (Ping timeout: 480 seconds)
[0:43] * Bartek (~Bartek@78.10.236.140) Quit (Ping timeout: 480 seconds)
[0:45] * espeer_ (~quassel@phobos.isoho.st) has joined #ceph
[0:45] * Skaag (~lunix@65.200.54.234) Quit (Quit: Leaving.)
[0:45] * davidzlap (~Adium@2605:e000:1313:8003:9c4e:b36e:c89a:79ca) Quit (Quit: Leaving.)
[0:45] * davidzlap (~Adium@2605:e000:1313:8003:9c4e:b36e:c89a:79ca) has joined #ceph
[0:46] * espeer (~quassel@phobos.isoho.st) Quit (Ping timeout: 480 seconds)
[0:47] * rakeshgm (~rakesh@106.51.225.4) Quit (Quit: Leaving)
[0:49] * Skaag (~lunix@65.200.54.234) has joined #ceph
[0:50] * Chrissi_ (~dontron@tor1e1.privacyfoundation.ch) has joined #ceph
[0:56] * angdraug (~angdraug@64.124.158.100) Quit (Quit: Leaving)
[1:03] * bjornar__ (~bjornar@ti0099a430-1561.bb.online.no) Quit (Ping timeout: 480 seconds)
[1:07] * Skaag (~lunix@65.200.54.234) Quit (Quit: Leaving.)
[1:12] * lcurtis_ (~lcurtis@47.19.105.250) Quit (Ping timeout: 480 seconds)
[1:14] * Racpatel (~Racpatel@2601:87:3:3601::4edb) Quit (Quit: Leaving)
[1:18] * shyu_ (~shyu@111.197.116.113) Quit (Ping timeout: 480 seconds)
[1:20] * boichev (~boichev@213.169.56.130) Quit (Ping timeout: 480 seconds)
[1:20] * Chrissi_ (~dontron@06SAAA1BZ.tor-irc.dnsbl.oftc.net) Quit ()
[1:20] * clusterfudge (~loft@atlantic480.us.unmetered.com) has joined #ceph
[1:21] * Skaag (~lunix@65.200.54.234) has joined #ceph
[1:33] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[1:34] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit ()
[1:35] * Lea (~LeaChim@host86-168-120-216.range86-168.btcentralplus.com) Quit (Ping timeout: 480 seconds)
[1:37] * cathode (~cathode@50.232.215.114) Quit (Quit: Leaving)
[1:40] * dgurtner_ (~dgurtner@217.149.140.193) Quit (Ping timeout: 480 seconds)
[1:42] * Skaag (~lunix@65.200.54.234) Quit (Quit: Leaving.)
[1:50] * clusterfudge (~loft@6AGAAAIV8.tor-irc.dnsbl.oftc.net) Quit ()
[1:50] * CoZmicShReddeR (~mps@4MJAAD1DF.tor-irc.dnsbl.oftc.net) has joined #ceph
[1:54] * oms101 (~oms101@p20030057EA04B100C6D987FFFE4339A1.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[1:54] * guerby (~guerby@2a03:7220:8080:a500::1) Quit (Ping timeout: 480 seconds)
[1:58] * b_rake (~b_rake@69-195-66-67.unifiedlayer.com) has joined #ceph
[2:03] * oms101 (~oms101@p20030057EA03A700C6D987FFFE4339A1.dip0.t-ipconnect.de) has joined #ceph
[2:05] * Kupo1 (~tyler.wil@23.111.254.159) has left #ceph
[2:05] * xarses_ (~xarses@64.124.158.100) Quit (Ping timeout: 480 seconds)
[2:06] * nwf (~nwf@00018577.user.oftc.net) has joined #ceph
[2:08] * winston-d_ (uid98317@id-98317.richmond.irccloud.com) has joined #ceph
[2:08] * bene2 (~bene@2601:18c:8501:25e4:ea2a:eaff:fe08:3c7a) Quit (Quit: Konversation terminated!)
[2:08] * Skaag (~lunix@65.200.54.234) has joined #ceph
[2:10] * wyang (~wyang@122.225.69.23) has joined #ceph
[2:10] * guerby (~guerby@ip165.tetaneutral.net) has joined #ceph
[2:10] * lx0 (~aoliva@lxo.user.oftc.net) has joined #ceph
[2:12] * wyang (~wyang@122.225.69.23) Quit ()
[2:15] * Skaag (~lunix@65.200.54.234) Quit (Quit: Leaving.)
[2:15] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[2:18] * RameshN (~rnachimu@101.222.248.37) has joined #ceph
[2:20] * CoZmicShReddeR (~mps@4MJAAD1DF.tor-irc.dnsbl.oftc.net) Quit ()
[2:25] * lx0 (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[2:34] * lx0 (~aoliva@lxo.user.oftc.net) has joined #ceph
[2:37] * dkrdkr1 (~oftc-webi@72.163.220.16) Quit (Ping timeout: 480 seconds)
[2:41] * DV_ (~veillard@2001:41d0:a:f29f::1) has joined #ceph
[2:43] * yguang11 (~yguang11@nat-dip27-wl-a.cfw-a-gci.corp.yahoo.com) Quit (Remote host closed the connection)
[2:43] * sudocat (~dibarra@192.185.1.20) Quit (Ping timeout: 480 seconds)
[2:50] * Doodlepieguy (~darkid@tesla16.play-with-crypto.space) has joined #ceph
[2:53] * georgem (~Adium@45.72.252.167) has joined #ceph
[3:13] * yanzheng (~zhyan@118.116.113.6) has joined #ceph
[3:20] * Doodlepieguy (~darkid@4MJAAD1FD.tor-irc.dnsbl.oftc.net) Quit ()
[3:20] * _br_ (~zc00gii@tor-exit-4.all.de) has joined #ceph
[3:29] * EinstCrazy (~EinstCraz@58.247.119.250) has joined #ceph
[3:30] * mhack (~mhack@66-168-117-78.dhcp.oxfr.ma.charter.com) Quit (Remote host closed the connection)
[3:32] * mikecao (~caobo@223.80.96.63) has joined #ceph
[3:34] * olid19811115 (~olid1982@aftr-185-17-206-43.dynamic.mnet-online.de) Quit (Ping timeout: 480 seconds)
[3:35] * derjohn_mobi (~aj@x4db1216c.dyn.telefonica.de) has joined #ceph
[3:40] * zhaochao (~zhaochao@125.39.9.156) has joined #ceph
[3:43] * derjohn_mob (~aj@x4db29d58.dyn.telefonica.de) Quit (Ping timeout: 480 seconds)
[3:50] * _br_ (~zc00gii@6AGAAAI1G.tor-irc.dnsbl.oftc.net) Quit ()
[3:50] * Misacorp (~loft@3.tor.exit.babylon.network) has joined #ceph
[3:55] * RameshN (~rnachimu@101.222.248.37) Quit (Ping timeout: 480 seconds)
[3:56] * Mika_c (~quassel@122.146.93.152) has joined #ceph
[3:59] * vicente (~~vicente@125-227-238-55.HINET-IP.hinet.net) has joined #ceph
[3:59] * RameshN (~rnachimu@101.222.248.37) has joined #ceph
[4:01] * vasu (~vasu@c-73-231-60-138.hsd1.ca.comcast.net) Quit (Quit: Leaving)
[4:12] * chasmo77 (~chas77@158.183-62-69.ftth.swbr.surewest.net) Quit (Ping timeout: 480 seconds)
[4:15] * Mika_c_ (~quassel@122.146.93.152) has joined #ceph
[4:17] * efirs (~firs@c-50-185-70-125.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[4:17] * wwdillingham (~LobsterRo@209-6-222-74.c3-0.hdp-ubr1.sbo-hdp.ma.cable.rcn.com) has joined #ceph
[4:20] * Misacorp (~loft@4MJAAD1HQ.tor-irc.dnsbl.oftc.net) Quit ()
[4:20] * measter (~Izanagi@torsrva.snydernet.net) has joined #ceph
[4:21] * lx0 (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[4:21] * wwdillingham (~LobsterRo@209-6-222-74.c3-0.hdp-ubr1.sbo-hdp.ma.cable.rcn.com) Quit ()
[4:21] * kefu (~kefu@183.193.163.144) has joined #ceph
[4:22] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[4:23] * Mika_c (~quassel@122.146.93.152) Quit (Ping timeout: 480 seconds)
[4:23] * krobelus_ (~krobelus@178-191-110-232.adsl.highway.telekom.at) Quit (Quit: leaving)
[4:25] * krobelus (~krobelus@178-191-110-232.adsl.highway.telekom.at) has joined #ceph
[4:26] * RameshN (~rnachimu@101.222.248.37) Quit (Ping timeout: 480 seconds)
[4:31] * mikecao (~caobo@223.80.96.63) Quit (Ping timeout: 480 seconds)
[4:35] * EinstCrazy (~EinstCraz@58.247.119.250) Quit (Remote host closed the connection)
[4:36] * EinstCrazy (~EinstCraz@58.247.119.250) has joined #ceph
[4:41] * kefu_ (~kefu@114.92.120.83) has joined #ceph
[4:42] * kefu (~kefu@183.193.163.144) Quit (Ping timeout: 480 seconds)
[4:44] * wwdillingham (~LobsterRo@209-6-222-74.c3-0.hdp-ubr1.sbo-hdp.ma.cable.rcn.com) has joined #ceph
[4:44] * Skaag (~lunix@cpe-172-91-77-84.socal.res.rr.com) has joined #ceph
[4:50] * measter (~Izanagi@6AGAAAI3V.tor-irc.dnsbl.oftc.net) Quit ()
[4:50] * Szernex (~bildramer@94.242.195.186) has joined #ceph
[4:52] * wwdillingham (~LobsterRo@209-6-222-74.c3-0.hdp-ubr1.sbo-hdp.ma.cable.rcn.com) Quit (Quit: wwdillingham)
[4:53] * wwdillingham (~LobsterRo@209-6-222-74.c3-0.hdp-ubr1.sbo-hdp.ma.cable.rcn.com) has joined #ceph
[5:00] * wwdillingham (~LobsterRo@209-6-222-74.c3-0.hdp-ubr1.sbo-hdp.ma.cable.rcn.com) Quit (Quit: wwdillingham)
[5:02] * xarses (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) has joined #ceph
[5:04] * overclk (~quassel@117.202.98.8) has joined #ceph
[5:05] * debian112 (~bcolbert@24.126.201.64) has left #ceph
[5:05] * wwdillingham (~LobsterRo@209-6-222-74.c3-0.hdp-ubr1.sbo-hdp.ma.cable.rcn.com) has joined #ceph
[5:10] * wwdillingham (~LobsterRo@209-6-222-74.c3-0.hdp-ubr1.sbo-hdp.ma.cable.rcn.com) Quit (Quit: wwdillingham)
[5:13] * Kesav (~oftc-webi@c-50-185-122-97.hsd1.ca.comcast.net) has joined #ceph
[5:19] <Kesav> Hi I'm very new to ceph. I'm using PostgreSQL as my database and wondering is ceph RBD a good fit for keeping PostgreSQL database? I tried using GlusterFS and PostgreSQL didn't work. The write frequency is way too much to handle for Gluster. My workload of database is I've almost 50GB of data. I run lot of ETL jobs at times where lots of data gets into the system. Non ETL time it's mostly powering my web application.
[5:19] <Kesav> Can any one share their experiences of using ceph for database?
[5:20] * Szernex (~bildramer@4MJAAD1JE.tor-irc.dnsbl.oftc.net) Quit ()
[5:20] * mikecao (~caobo@223.80.96.63) has joined #ceph
[5:26] * Vacuum__ (~Vacuum@i59F79C79.versanet.de) has joined #ceph
[5:27] * Kesav (~oftc-webi@c-50-185-122-97.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[5:33] * Vacuum_ (~Vacuum@88.130.217.241) Quit (Ping timeout: 480 seconds)
[5:34] <[arx]> left before we could link: https://www.youtube.com/watch?v=OqlC7S3cUKs
[5:37] * efirs (~firs@c-50-185-70-125.hsd1.ca.comcast.net) has joined #ceph
[5:38] * shyu_ (~shyu@119.254.120.71) has joined #ceph
[5:43] * shyu (~Shanzhi@119.254.120.66) Quit (Ping timeout: 480 seconds)
[5:46] * kefu_ (~kefu@114.92.120.83) Quit (Max SendQ exceeded)
[5:46] * kefu (~kefu@114.92.120.83) has joined #ceph
[5:56] * georgem (~Adium@45.72.252.167) Quit (Quit: Leaving.)
[6:00] * jamespage (~jamespage@2a00:1098:0:80:1000:42:0:1) Quit (Quit: Coyote finally caught me)
[6:01] * jamespage (~jamespage@culvain.gromper.net) has joined #ceph
[6:18] * vikhyat (~vumrao@121.244.87.116) has joined #ceph
[6:26] * RameshN (~rnachimu@121.244.87.117) has joined #ceph
[6:29] * wjw-freebsd (~wjw@smtp.digiware.nl) Quit (Ping timeout: 480 seconds)
[6:38] * swami1 (~swami@49.44.57.245) has joined #ceph
[6:39] * efirs (~firs@c-50-185-70-125.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[6:50] * Thayli (~Diablothe@192.42.115.101) has joined #ceph
[6:54] * Hemanth (~hkumar_@103.228.221.173) has joined #ceph
[7:03] * efirs (~firs@c-50-185-70-125.hsd1.ca.comcast.net) has joined #ceph
[7:04] * yguang11 (~yguang11@c-73-189-35-7.hsd1.ca.comcast.net) has joined #ceph
[7:09] * kawa2014 (~kawa@94.161.229.174) has joined #ceph
[7:16] * caobo_ (~caobo@223.80.96.63) has joined #ceph
[7:16] * mikecao (~caobo@223.80.96.63) Quit (Read error: Connection reset by peer)
[7:20] * Thayli (~Diablothe@4MJAAD1MR.tor-irc.dnsbl.oftc.net) Quit ()
[7:20] * overclk (~quassel@117.202.98.8) Quit (Ping timeout: 480 seconds)
[7:20] * Grum (~dicko@65.19.167.132) has joined #ceph
[7:22] * khyron (~khyron@187.190.152.61) has joined #ceph
[7:30] * mikecao (~caobo@223.80.96.63) has joined #ceph
[7:30] * caobo_ (~caobo@223.80.96.63) Quit (Read error: Connection reset by peer)
[7:41] * swami1 (~swami@49.44.57.245) Quit (Ping timeout: 480 seconds)
[7:46] * BrianA (~Brian@c-50-168-46-112.hsd1.ca.comcast.net) has joined #ceph
[7:46] * Concubidated (~cube@c-50-173-245-118.hsd1.ca.comcast.net) Quit (Read error: Connection reset by peer)
[7:46] * Concubidated (~cube@c-50-173-245-118.hsd1.ca.comcast.net) has joined #ceph
[7:46] <BrianA> anyone familiar with upgrading ceph from hammer to infernalis on CentOS 7?
[7:50] * Grum (~dicko@4MJAAD1NM.tor-irc.dnsbl.oftc.net) Quit ()
[7:50] * `Jin (~K3NT1S_aw@86.104.15.15) has joined #ceph
[7:50] * shyu (~Shanzhi@119.254.120.66) has joined #ceph
[7:53] * yguang11 (~yguang11@c-73-189-35-7.hsd1.ca.comcast.net) Quit (Remote host closed the connection)
[7:55] * swami1 (~swami@49.32.0.112) has joined #ceph
[7:57] * overclk (~quassel@121.244.87.124) has joined #ceph
[7:57] * nils_ (~nils_@doomstreet.collins.kg) has joined #ceph
[8:02] * overclk_ (~quassel@121.244.87.117) has joined #ceph
[8:05] * chasmo77 (~chas77@158.183-62-69.ftth.swbr.surewest.net) has joined #ceph
[8:05] * overclk (~quassel@121.244.87.124) Quit (Ping timeout: 480 seconds)
[8:08] * karnan (~karnan@121.244.87.117) has joined #ceph
[8:08] * Be-El (~blinke@nat-router.computational.bio.uni-giessen.de) has joined #ceph
[8:08] * krobelus_ (~krobelus@193-83-183-66.adsl.highway.telekom.at) has joined #ceph
[8:09] * branto (~branto@nat-pool-brq-t.redhat.com) has joined #ceph
[8:15] * krobelus (~krobelus@178-191-110-232.adsl.highway.telekom.at) Quit (Ping timeout: 480 seconds)
[8:20] * `Jin (~K3NT1S_aw@06SAAA1PC.tor-irc.dnsbl.oftc.net) Quit ()
[8:20] * Malcovent (~redbeast1@06SAAA1QB.tor-irc.dnsbl.oftc.net) has joined #ceph
[8:26] * reed (~reed@75-101-54-18.dsl.static.fusionbroadband.com) Quit (Ping timeout: 480 seconds)
[8:26] * reed (~reed@75-101-54-18.dsl.static.fusionbroadband.com) has joined #ceph
[8:33] * Miouge (~Miouge@h-72-233.a163.priv.bahnhof.se) has joined #ceph
[8:38] * shyu_ (~shyu@119.254.120.71) Quit (Quit: Leaving)
[8:39] * shyu (~Shanzhi@119.254.120.66) Quit (Quit: Leaving)
[8:39] * shyu (~Shanzhi@119.254.120.66) has joined #ceph
[8:45] * mlovell (~mlovell@69-195-66-94.unifiedlayer.com) Quit (Ping timeout: 480 seconds)
[8:48] * efirs (~firs@c-50-185-70-125.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[8:50] * Malcovent (~redbeast1@06SAAA1QB.tor-irc.dnsbl.oftc.net) Quit ()
[8:50] * zviratko (~TehZomB@tor.exit-no.de) has joined #ceph
[8:50] * kawa2014 (~kawa@94.161.229.174) Quit (Quit: Leaving)
[8:59] * krypto (~krypto@125.16.137.146) has joined #ceph
[8:59] * EinstCra_ (~EinstCraz@58.247.119.250) has joined #ceph
[8:59] * fsimonce (~simon@host201-70-dynamic.26-79-r.retail.telecomitalia.it) has joined #ceph
[9:01] * rendar (~I@95.234.180.115) has joined #ceph
[9:02] * dgurtner (~dgurtner@82.199.64.68) has joined #ceph
[9:03] * T1w (~jens@217.195.184.71) has joined #ceph
[9:04] * mlovell (~mlovell@2605:a601:6e1:1100:3164:de0:2b2d:a463) has joined #ceph
[9:06] * Hemanth (~hkumar_@103.228.221.173) Quit (Ping timeout: 480 seconds)
[9:06] * EinstCrazy (~EinstCraz@58.247.119.250) Quit (Ping timeout: 480 seconds)
[9:06] * analbeard (~shw@support.memset.com) has joined #ceph
[9:10] * Be-El (~blinke@nat-router.computational.bio.uni-giessen.de) Quit (Quit: Leaving.)
[9:13] * Be-El (~blinke@nat-router.computational.bio.uni-giessen.de) has joined #ceph
[9:19] * Hemanth (~hkumar_@103.228.221.173) has joined #ceph
[9:20] * zviratko (~TehZomB@06SAAA1Q9.tor-irc.dnsbl.oftc.net) Quit ()
[9:20] * Pommesgabel (~colde@94.242.195.186) has joined #ceph
[9:22] * wjw-freebsd (~wjw@smtp.digiware.nl) has joined #ceph
[9:23] * allaok (~allaok@machine107.orange-labs.com) has joined #ceph
[9:24] * olid19811115 (~olid1982@aftr-185-17-206-247.dynamic.mnet-online.de) has joined #ceph
[9:26] * kefu (~kefu@114.92.120.83) Quit (Read error: Connection reset by peer)
[9:26] * kefu (~kefu@114.92.120.83) has joined #ceph
[9:32] * b_rake (~b_rake@69-195-66-67.unifiedlayer.com) Quit (Ping timeout: 480 seconds)
[9:32] * derjohn_mobi (~aj@x4db1216c.dyn.telefonica.de) Quit (Ping timeout: 480 seconds)
[9:41] * kmroz (~kilo@00020103.user.oftc.net) Quit (Quit: WeeChat 1.4)
[9:45] * b_rake (~b_rake@2605:a601:579:1f01:ec94:836f:a92:b529) has joined #ceph
[9:47] * kmroz (~kilo@00020103.user.oftc.net) has joined #ceph
[9:48] * b_rake (~b_rake@2605:a601:579:1f01:ec94:836f:a92:b529) Quit (Remote host closed the connection)
[9:48] * kefu (~kefu@114.92.120.83) Quit (Max SendQ exceeded)
[9:48] * bara (~bara@nat-pool-brq-t.redhat.com) has joined #ceph
[9:49] * kefu (~kefu@114.92.120.83) has joined #ceph
[9:50] * Pommesgabel (~colde@6AGAAAJFM.tor-irc.dnsbl.oftc.net) Quit ()
[9:50] * caobo_ (~caobo@223.80.96.63) has joined #ceph
[9:50] * ade (~abradshaw@212.77.58.61) has joined #ceph
[9:50] * roaet (~Defaultti@8.tor.exit.babylon.network) has joined #ceph
[9:51] * Geph (~Geoffrey@41.77.153.99) has joined #ceph
[9:52] * TMM (~hp@178-84-46-106.dynamic.upc.nl) Quit (Quit: Ex-Chat)
[9:57] * mikecao (~caobo@223.80.96.63) Quit (Ping timeout: 480 seconds)
[9:59] * ade (~abradshaw@212.77.58.61) Quit (Ping timeout: 480 seconds)
[10:00] * DanFoster (~Daniel@2a00:1ee0:3:1337:52b:8f9e:3178:1e1d) has joined #ceph
[10:03] * rotbeard (~redbeard@185.32.80.238) has joined #ceph
[10:03] <analbeard> morning all
[10:04] <analbeard> has anyone running Trusty seen it fail to automount the osds on boot?
[10:04] * farblue (~farblue@wor.namesco.net) has joined #ceph
[10:04] <analbeard> the ceph-disk activate traceback shows "--default', 'sysvinit" which isn't going to work
[10:06] * jordanP (~jordan@204.13-14-84.ripe.coltfrance.com) has joined #ceph
[10:06] * jowilkin (~jowilkin@c-50-148-138-174.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[10:09] * kawa2014 (~kawa@89.184.114.246) has joined #ceph
[10:11] * ade (~abradshaw@212.77.58.61) has joined #ceph
[10:16] <analbeard> sussed it - Infernalis needs python-setuptools
[10:20] * roaet (~Defaultti@6AGAAAJGW.tor-irc.dnsbl.oftc.net) Quit ()
[10:20] * bjornar__ (~bjornar@109.247.131.38) has joined #ceph
[10:31] * dyasny (~dyasny@bzq-82-81-161-51.red.bezeqint.net) has joined #ceph
[10:31] * ade (~abradshaw@212.77.58.61) Quit (Quit: Too sexy for his shirt)
[10:32] * rraja (~rraja@121.244.87.117) has joined #ceph
[10:33] * ngoswami (~ngoswami@121.244.87.116) has joined #ceph
[10:35] * ngoswami (~ngoswami@121.244.87.116) Quit ()
[10:35] * derjohn_mobi (~aj@2001:6f8:1337:0:ad23:e347:ed38:ee36) has joined #ceph
[10:43] * Lea (~LeaChim@host86-159-239-193.range86-159.btcentralplus.com) has joined #ceph
[10:45] * rmart04 (~rmart04@support.memset.com) has joined #ceph
[10:47] * TMM (~hp@185.5.122.2) has joined #ceph
[10:50] * b_rake (~b_rake@45.56.5.110) has joined #ceph
[10:50] * MonkeyJamboree (~Jourei@06SAAA1VW.tor-irc.dnsbl.oftc.net) has joined #ceph
[10:51] * karnan (~karnan@121.244.87.117) Quit (Remote host closed the connection)
[10:52] * Miouge_ (~Miouge@94.136.92.20) has joined #ceph
[10:55] <Miouge_> In the architecure doc I can???t find info on how the write acknolwedgement works for EC pools. Does it wait for all chunks to be in journal like for replicated pools?
[10:55] * ingard (~cake@tu.rd.vc) has joined #ceph
[10:56] * inv (~inv@windows.freehost.com.ua) has joined #ceph
[10:57] <inv> hi people
[11:01] * karnan (~karnan@121.244.87.117) has joined #ceph
[11:05] * rakeshgm (~rakesh@121.244.87.117) has joined #ceph
[11:06] * inv (~inv@windows.freehost.com.ua) Quit (Quit: ?????????? ?? ???? ?????? (xchat 2.4.5 ?????? ????????????))
[11:16] * evelu (~erwan@46.231.131.178) has joined #ceph
[11:17] <loicd> ingard: I don't know enough about the state of your cluster to advise your on how to repair it, sorry :-/
[11:18] * krobelus_ (~krobelus@193-83-183-66.adsl.highway.telekom.at) has left #ceph
[11:18] <etienneme> You maybe met this "issue" http://tracker.ceph.com/issues/12782
[11:19] <ingard> Bosse: ^
[11:19] <loicd> Bosse: I don't have much experience in cluster administration. If you're looking into fixing your cluster quickly, others are probably better at this.
[11:19] <ingard> loicd: others are not responding :P
[11:20] * MonkeyJamboree (~Jourei@06SAAA1VW.tor-irc.dnsbl.oftc.net) Quit ()
[11:20] <loicd> ingard: others are maybe busy or asleep :-)
[11:20] * `Jin (~Xeon06@91.219.236.222) has joined #ceph
[11:20] <loicd> speaking of which, it's time for coffee
[11:21] <ingard> i'd say breaking backwards compatability in a point release on lts is grounds for disturbing them
[11:24] <etienneme> I think you can still add straw alg osd
[11:25] <ingard> etienneme: as in readding the osds with forcing straw algo ?
[11:26] <etienneme> I mean I don't think it breaks compatibility as you can still add compatible OSD.
[11:27] * b_rake (~b_rake@45.56.5.110) Quit (Remote host closed the connection)
[11:28] <ingard> as long as a default is changed it will break
[11:29] <ingard> ^ if that is the case obviously
[11:29] * ngoswami (~ngoswami@121.244.87.116) has joined #ceph
[11:36] * zhaochao_ (~zhaochao@125.39.9.144) has joined #ceph
[11:37] * RameshN (~rnachimu@121.244.87.117) Quit (Ping timeout: 480 seconds)
[11:41] * zhaochao (~zhaochao@125.39.9.156) Quit (Ping timeout: 480 seconds)
[11:41] * zhaochao_ is now known as zhaochao
[11:48] * RameshN (~rnachimu@121.244.87.117) has joined #ceph
[11:49] <Bosse> loicd: http://tracker.ceph.com/issues/15411
[11:50] * `Jin (~Xeon06@6AGAAAJKI.tor-irc.dnsbl.oftc.net) Quit ()
[11:50] * Sliker (~skrblr@torsrva.snydernet.net) has joined #ceph
[11:51] <etienneme> You can have some straw and straw2 alg on the same cluster
[11:52] <Bosse> we tried removing the OSDs that was introduced with 0.94.6, but our RBD clients still derp out with a feature set mismatch referring to CEPH_FEATURE_CRUSH_V4
[11:52] <etienneme> But you maybe have some ceph component that can't handle this alg yet
[11:53] <etienneme> You decompiled the crush map to check there is no more straw2?
[11:54] <ingard> only mentioning of straw2 is now under the new hosts themselves (which no longer has any osds)
[11:56] <etienneme> You could try to remove them
[11:56] <etienneme> But really not sure of the result
[11:56] * jowilkin (~jowilkin@c-98-207-136-41.hsd1.ca.comcast.net) has joined #ceph
[11:59] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[12:00] * yanzheng (~zhyan@118.116.113.6) Quit (Quit: This computer has gone to sleep)
[12:01] <Bosse> ok, i decompiled the crushmap, removed the host entries and injected it back. RBD is now online again :)
[12:01] <Bosse> thanks for your help loicd and etienneme
[12:01] <etienneme> I'm happy it worked ;)
[12:06] * ngoswami (~ngoswami@121.244.87.116) Quit (Read error: Connection reset by peer)
[12:08] * ngoswami (~ngoswami@121.244.87.116) has joined #ceph
[12:08] * derjohn_mobi (~aj@2001:6f8:1337:0:ad23:e347:ed38:ee36) Quit (Ping timeout: 480 seconds)
[12:13] * krobelus (~krobelus@193-83-183-66.adsl.highway.telekom.at) has joined #ceph
[12:13] * krobelus (~krobelus@193-83-183-66.adsl.highway.telekom.at) Quit ()
[12:14] * krobelus (~krobelus@193-83-183-66.adsl.highway.telekom.at) has joined #ceph
[12:14] * krobelus (~krobelus@193-83-183-66.adsl.highway.telekom.at) Quit ()
[12:19] * overclk_ (~quassel@121.244.87.117) Quit (Remote host closed the connection)
[12:20] * Sliker (~skrblr@4MJAAD1UW.tor-irc.dnsbl.oftc.net) Quit ()
[12:23] * fdmanana (~fdmanana@2001:8a0:6e0c:6601:60fc:c21d:c1c6:96e4) has joined #ceph
[12:25] * Scaevolus (~airsoftgl@176.123.2.35) has joined #ceph
[12:28] * ira (~ira@c-24-34-255-34.hsd1.ma.comcast.net) has joined #ceph
[12:29] <loicd> Bosse: thanks for reporting the issue :-)
[12:30] * EinstCra_ (~EinstCraz@58.247.119.250) Quit (Remote host closed the connection)
[12:30] * RameshN (~rnachimu@121.244.87.117) Quit (Quit: Quit)
[12:30] * RameshN (~rnachimu@121.244.87.117) has joined #ceph
[12:34] * kefu (~kefu@114.92.120.83) Quit (Max SendQ exceeded)
[12:34] * kefu (~kefu@114.92.120.83) has joined #ceph
[12:44] * koma (~koma@0001c112.user.oftc.net) Quit (Quit: ZNC - http://znc.in)
[12:44] * TMM (~hp@185.5.122.2) Quit (Quit: Ex-Chat)
[12:48] * kefu is now known as kefu|afk
[12:51] <ingard> loicd: etienneme yeah thanks for assisting. unfortunately the fs on the rbd mount is corrupted :s
[12:52] * winston-d_ (uid98317@id-98317.richmond.irccloud.com) Quit (Quit: Connection closed for inactivity)
[12:54] * caobo_ (~caobo@223.80.96.63) Quit (Ping timeout: 480 seconds)
[12:54] * kefu|afk (~kefu@114.92.120.83) Quit (Max SendQ exceeded)
[12:54] * wjw-freebsd (~wjw@smtp.digiware.nl) Quit (Ping timeout: 480 seconds)
[12:55] * Scaevolus (~airsoftgl@06SAAA1YF.tor-irc.dnsbl.oftc.net) Quit ()
[12:55] * Kidlvr (~tallest_r@188.214.129.85) has joined #ceph
[12:55] * kefu (~kefu@114.92.120.83) has joined #ceph
[13:02] <loicd> ingard: it's really strange
[13:03] * bara (~bara@nat-pool-brq-t.redhat.com) Quit (Quit: Bye guys! (??????????????????? ?????????)
[13:06] <ingard> loicd: how is it strange?
[13:07] <loicd> that it's corrupted
[13:07] <ingard> i agree. but it could be that it got corrupted when the fs "went away" after the upgrade
[13:07] <ingard> maybe there was some unflushed changes or whatever
[13:08] * Mika_c_ (~quassel@122.146.93.152) Quit (Remote host closed the connection)
[13:09] <etienneme> You have some inconsistent pg? Or other errors?
[13:11] * overclk (~quassel@117.202.104.103) has joined #ceph
[13:12] * vicente (~~vicente@125-227-238-55.HINET-IP.hinet.net) Quit (Quit: Leaving)
[13:14] * bniver (~bniver@pool-173-48-58-27.bstnma.fios.verizon.net) Quit (Remote host closed the connection)
[13:18] * nils_ (~nils_@doomstreet.collins.kg) Quit (Ping timeout: 480 seconds)
[13:22] * wjw-freebsd (~wjw@176.74.240.1) has joined #ceph
[13:24] * Racpatel (~Racpatel@2601:87:3:3601::2bdf) has joined #ceph
[13:25] * Kidlvr (~tallest_r@4MJAAD1WD.tor-irc.dnsbl.oftc.net) Quit ()
[13:25] * nastidon (~Mraedis@4MJAAD1W0.tor-irc.dnsbl.oftc.net) has joined #ceph
[13:33] * zhaochao (~zhaochao@125.39.9.144) Quit (Quit: ChatZilla 0.9.92 [Firefox 45.0.1/20160318172635])
[13:34] <Bosse> etienneme: no other errors than the backfills after we removed the new OSDs
[13:35] <Bosse> so, at first glance the fs seemed corrupted, but when I'm trying to remove a test RBD volume, it hangs with no information in the logs. i suspect the problem isn't completely gone.
[13:38] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit (Quit: Leaving.)
[13:54] * karnan (~karnan@121.244.87.117) Quit (Quit: Leaving)
[13:54] * wyang (~wyang@114.111.166.46) has joined #ceph
[13:54] * branto (~branto@nat-pool-brq-t.redhat.com) Quit (Quit: Leaving.)
[13:55] * bniver (~bniver@71-9-144-29.static.oxfr.ma.charter.com) has joined #ceph
[13:55] * nastidon (~Mraedis@4MJAAD1W0.tor-irc.dnsbl.oftc.net) Quit ()
[13:59] * nils_ (~nils_@doomstreet.collins.kg) has joined #ceph
[14:04] * nils_ (~nils_@doomstreet.collins.kg) Quit (Quit: This computer has gone to sleep)
[14:06] * EinstCrazy (~EinstCraz@58.39.19.226) has joined #ceph
[14:20] * mikecao (~caobo@123.235.20.194) has joined #ceph
[14:20] * wyang (~wyang@114.111.166.46) Quit (Quit: This computer has gone to sleep)
[14:23] * nils_ (~nils_@doomstreet.collins.kg) has joined #ceph
[14:31] * mattbenjamin1 (~mbenjamin@76-206-42-50.lightspeed.livnmi.sbcglobal.net) has joined #ceph
[14:36] * joshd1 (~jdurgin@68-119-140-18.dhcp.ahvl.nc.charter.com) has joined #ceph
[14:38] * wjw-freebsd2 (~wjw@176.74.240.1) has joined #ceph
[14:40] * i_m (~ivan.miro@deibp9eh1--blueice3n0.emea.ibm.com) has joined #ceph
[14:41] * wjw-freebsd (~wjw@176.74.240.1) Quit (Ping timeout: 480 seconds)
[14:50] * georgem (~Adium@206.108.127.16) has joined #ceph
[14:51] * wjw-freebsd2 (~wjw@176.74.240.1) Quit (Ping timeout: 480 seconds)
[14:52] * wjw-freebsd (~wjw@176.74.240.1) has joined #ceph
[14:54] * yanzheng (~zhyan@118.116.113.6) has joined #ceph
[14:57] * thomnico (~thomnico@37.162.121.55) has joined #ceph
[14:57] * overclk_ (~quassel@117.202.101.187) has joined #ceph
[14:58] * wyang (~wyang@114.111.166.46) has joined #ceph
[14:59] * mhack (~mhack@nat-pool-bos-t.redhat.com) has joined #ceph
[14:59] * Hemanth (~hkumar_@103.228.221.173) Quit (Ping timeout: 480 seconds)
[14:59] * caobo_ (~caobo@123.235.20.194) has joined #ceph
[14:59] * overclk (~quassel@117.202.104.103) Quit (Ping timeout: 480 seconds)
[15:00] * mikecao (~caobo@123.235.20.194) Quit (Ping timeout: 480 seconds)
[15:02] * rwheeler (~rwheeler@pool-173-48-195-215.bstnma.fios.verizon.net) Quit (Quit: Leaving)
[15:03] * wwdillingham (~LobsterRo@mobile-166-186-169-85.mycingular.net) has joined #ceph
[15:03] * wwdillingham (~LobsterRo@mobile-166-186-169-85.mycingular.net) Quit ()
[15:04] * krypto (~krypto@125.16.137.146) Quit (Quit: Leaving)
[15:04] * whatevsz (~quassel@ip5f5adec4.dynamic.kabel-deutschland.de) has joined #ceph
[15:07] * shyu_ (~shyu@111.197.116.113) has joined #ceph
[15:07] * wyang (~wyang@114.111.166.46) Quit (Quit: This computer has gone to sleep)
[15:07] * RameshN (~rnachimu@121.244.87.117) Quit (Ping timeout: 480 seconds)
[15:08] * wyang (~wyang@114.111.166.46) has joined #ceph
[15:10] * rakeshgm (~rakesh@121.244.87.117) Quit (Quit: Leaving)
[15:15] * thomnico (~thomnico@37.162.121.55) Quit (Quit: Ex-Chat)
[15:16] * CoMa (~FierceFor@tor.ohundred.com) has joined #ceph
[15:16] * wyang (~wyang@114.111.166.46) Quit (Quit: This computer has gone to sleep)
[15:16] * wjw-freebsd (~wjw@176.74.240.1) Quit (Ping timeout: 480 seconds)
[15:17] * RameshN (~rnachimu@121.244.87.117) has joined #ceph
[15:20] * dneary (~dneary@nat-pool-bos-u.redhat.com) Quit (Ping timeout: 480 seconds)
[15:21] * overclk (~quassel@117.202.101.33) has joined #ceph
[15:23] * derjohn_mobi (~aj@2001:6f8:1337:0:816c:990a:d237:3b1c) has joined #ceph
[15:24] * overclk_ (~quassel@117.202.101.187) Quit (Ping timeout: 480 seconds)
[15:26] * wjw-freebsd (~wjw@176.74.240.1) has joined #ceph
[15:29] * bene2 (~bene@50.153.135.131) has joined #ceph
[15:32] * brad_mssw (~brad@66.129.88.50) has joined #ceph
[15:33] * wyang (~wyang@114.111.166.46) has joined #ceph
[15:37] * wwdillingham (~LobsterRo@65.112.8.195) has joined #ceph
[15:38] * mlovell (~mlovell@2605:a601:6e1:1100:3164:de0:2b2d:a463) Quit (Quit: Leaving.)
[15:39] * wjw-freebsd2 (~wjw@176.74.240.1) has joined #ceph
[15:40] * wjw-freebsd (~wjw@176.74.240.1) Quit (Ping timeout: 480 seconds)
[15:41] * RameshN (~rnachimu@121.244.87.117) Quit (Ping timeout: 480 seconds)
[15:42] * mattbenjamin1 (~mbenjamin@76-206-42-50.lightspeed.livnmi.sbcglobal.net) Quit (Quit: Leaving.)
[15:45] * wjw-freebsd2 (~wjw@176.74.240.1) Quit (Read error: Connection reset by peer)
[15:46] * CoMa (~FierceFor@4MJAAD100.tor-irc.dnsbl.oftc.net) Quit ()
[15:46] * Phase (~lobstar@tor-relay.zwiebeltoralf.de) has joined #ceph
[15:47] * Phase is now known as Guest10630
[15:47] * dneary (~dneary@pool-96-237-170-97.bstnma.fios.verizon.net) has joined #ceph
[15:49] * RameshN (~rnachimu@121.244.87.117) has joined #ceph
[15:53] * wjw-freebsd (~wjw@176.74.240.1) has joined #ceph
[15:54] * yanzheng (~zhyan@118.116.113.6) Quit (Quit: This computer has gone to sleep)
[16:00] * rwheeler (~rwheeler@nat-pool-bos-t.redhat.com) has joined #ceph
[16:01] * T1w (~jens@217.195.184.71) Quit (Ping timeout: 480 seconds)
[16:02] * MrAbaddon (~MrAbaddon@193.137.26.66) has joined #ceph
[16:02] <rmart04> Hi, I have a ceph pool showing some very strange data, has anyone seen this before?
[16:02] <rmart04> NAME ID USED %USED MAX AVAIL OBJECTS
[16:02] <rmart04> volumes_p02 3 8E 0 53944G -9223372036854751074
[16:03] <etienneme> No :|
[16:04] <etienneme> You could create an issue, you did nothing weird?
[16:04] <rmart04> Not that I know of, we increased the pg count to 2048 a couple of days ago
[16:04] <rmart04> thats all
[16:04] * bene3 (~bene@2601:18c:8501:25e4:ea2a:eaff:fe08:3c7a) has joined #ceph
[16:05] * TMM (~hp@185.5.122.2) has joined #ceph
[16:08] * bene2 (~bene@50.153.135.131) Quit (Ping timeout: 480 seconds)
[16:09] * thomnico (~thomnico@132.217.26.109.rev.sfr.net) has joined #ceph
[16:09] <etienneme> I think only dev could help you. I don't know how are counted the objects.
[16:12] <rmart04> OK, i can list the contents of the pool, its just like the overview of the cluster is broken, and as such I Dont think its letting me create any new volumes as it thinks its full
[16:13] <rmart04> Actually it appears I can create images inside the ???corrupted??? pool too
[16:13] <BrianA> anyone familiar with upgrading ceph from hammer to infernalis on CentOS 7?
[16:14] * ade (~abradshaw@212.77.58.61) has joined #ceph
[16:16] * Guest10630 (~lobstar@06SAAA16Z.tor-irc.dnsbl.oftc.net) Quit ()
[16:16] * Grum (~Vale@edwardsnowden1.torservers.net) has joined #ceph
[16:16] <analbeard> @briana: I've recently upgraded a Trusty cluster from Hammer to Infernalis so I may be able to help (if it's not Centos specific)
[16:18] * whatevsz (~quassel@ip5f5adec4.dynamic.kabel-deutschland.de) Quit (Ping timeout: 480 seconds)
[16:18] * RameshN (~rnachimu@121.244.87.117) Quit (Quit: Quit)
[16:22] <TMM> I just had a host with 8 OSDs fail on me, and I ended up seeing slow requests of up to 240 seconds while the osds were still marked up.
[16:22] * wjw-freebsd2 (~wjw@176.74.240.1) has joined #ceph
[16:22] <TMM> when the monitor finally kicked all osds out of the cluster all slow requests stopped
[16:23] <TMM> is there a way to speed up this process a little? I have virtual instances running with scsi timeouts of 30 seconds
[16:23] <rmart04> This is where it freaks out for anyone who???s following : 2016-04-06 20:40:09.825610 mon.0 10.2.49.11:6789/0 2342689 : cluster [INF] pgmap v8662220: 8640 pgs: 8640 active+clean; 8 EB data, 5093 GB used, 160 TB / 165 TB avail; 0 B/s rd, 23919 B/s wr, 5 op/s
[16:23] <rmart04> 2016-04-06 20:40:10.851154 mon.0 10.2.49.11:6789/0 2342691 : cluster [INF] pgmap v8662221: 8640 pgs: 1 active+clean+scrubbing, 8639 active+clean; 8 EB data, 5093 GB used, 160 TB / 165 TB avail
[16:23] <TMM> ideally I'd have this whole process of 'osd seems bad, kill it' happen within the span of 30 seconds. Or failing that, having OSDs finish their transactions when it can't write just one of it's replicas
[16:24] <TMM> oh, it actually went up to 481 seconds :-/
[16:25] <TMM> Does anyone have any recommendations for configuration settings I should be looking at to optimize this a bit?
[16:25] <analbeard> TMM: you have a few different things which can influence that. mon_osd_down_out_interval, mon_osd_min_down_reporters, mon_osd_min_down_reports
[16:27] * wjw-freebsd (~wjw@176.74.240.1) Quit (Ping timeout: 480 seconds)
[16:28] <TMM> analbeard, thanks
[16:31] <TMM> Does anyone know if my conclusion "OSDs were down, other OSDs were waiting for requests to it to complete." Is a likely reason for the slow requests?
[16:33] * Miouge (~Miouge@h-72-233.a163.priv.bahnhof.se) Quit (Quit: Miouge)
[16:33] * Miouge_ is now known as Miouge
[16:34] <BrianA> @analbeard: I've started doing an upgrade following http://docs.ceph.com/docs/infernalis/install/upgrading-ceph/
[16:34] <BrianA> So far I have only replaced ceph-release and then run "ceph-deploy install --release infernalis" on the mons.
[16:34] <BrianA> The next steps are not documented for systemd, so I have to figure it out.
[16:34] <BrianA> The odd thing for me is that on the first mon, I have /usr/lib/systemd/system/ceph-mon@service but those are not there on the others.
[16:34] <BrianA> There is a service running that is ceph-mon.<hostname>.<some numbers> on all three and that is what I restarted since I could not restart ceph-mon@<hostname>
[16:36] * vikhyat (~vumrao@121.244.87.116) Quit (Quit: Leaving)
[16:37] * jproulx (~jon@kvas.csail.mit.edu) has joined #ceph
[16:37] <BrianA> Shouldn't I be doing it as 'systemctl restart ceph-mon@<hostname>'? I haven't restarted the OSDs because it looks similar for them.
[16:38] <rmart04> Has anyone used ???ceph sync force??? before?
[16:38] <analbeard> @briana: I've not used ceph with systemd, but youare correct that you should be doing ceph-mon@monname
[16:40] <analbeard> unless you need to provide the cluster name too
[16:40] <analbeard> so cluster-monname
[16:40] <BrianA> Hmm. when I try that I get an error:
[16:40] <BrianA> IO error: /var/lib/ceph/mon/ceph-borg01/store.db/LOCK: Permission denied
[16:40] <Kdecherf> can we segregate rbd volumes? (they are in the same pool)
[16:41] <analbeard> Briana: what's the ownership of /var/lib/ceph/mon/ceph-borg01/ and the files inside it?
[16:42] * markednmbr1 (~Diego@cpc1-lewi13-2-0-cust267.2-4.cable.virginm.net) has joined #ceph
[16:43] <markednmbr1> hello folks
[16:44] <markednmbr1> i've been playing around with a cachetier on ssd partition of each node to see if it helps performance
[16:44] <markednmbr1> I just rebooted all my machines after upgrading to 9.2 but for some reason the osds in the cache tier didnt come back on by themselves, I had to ceph-osd activate them
[16:45] <markednmbr1> now I've got "HEALTH_WARN 128 pgs stale; 128 pgs stuck stale"
[16:45] <markednmbr1> all on the cache tier osds
[16:46] * Grum (~Vale@06SAAA175.tor-irc.dnsbl.oftc.net) Quit ()
[16:46] * Revo84 (~Throlkim@6AGAAAJZY.tor-irc.dnsbl.oftc.net) has joined #ceph
[16:46] <BrianA> it is owned root:root and has done,keyring,store.db and sysvinit
[16:47] * Kurt (~Adium@2001:628:1:5:49fd:1c10:5b19:4293) Quit (Quit: Leaving.)
[16:48] * kawa2014 (~kawa@89.184.114.246) Quit (Ping timeout: 480 seconds)
[16:49] <analbeard> briana: infernalis runs as an unprivileged user ceph (unless you specifically said otherwise)
[16:49] <analbeard> https://ceph.com/releases/v9-2-0-infernalis-released/
[16:49] <analbeard> there are some bits on there you'll definitely find useful
[16:50] <BrianA> Yep..
[16:50] <BrianA> I changed the ownership of /var/lib/ceph/mon/ceph-borg01/store.db/ to ceph:ceph and now the mon starts as expected. I'll dig deeper to see if there is more I need to do before going on to the others.
[16:51] <nils_> any thoughts on tuning filestore max sync interval and min sync interval?
[16:51] <analbeard> briana: they suggest chowning the whole of /var/lib/ceph/, presumably to be on the safe side
[16:52] * rraja (~rraja@121.244.87.117) Quit (Ping timeout: 480 seconds)
[16:52] <BrianA> Cool. I need to look into the OSDs too, I see no ceph-osd@ on any of them but I do see many ceph-osd.<n>.<some numbers>
[17:01] * overclk_ (~quassel@117.202.96.251) has joined #ceph
[17:01] * kawa2014 (~kawa@5.90.37.231) has joined #ceph
[17:02] * whatevsz (~quassel@ip5f5adec4.dynamic.kabel-deutschland.de) has joined #ceph
[17:03] * dgurtner_ (~dgurtner@82.199.64.68) has joined #ceph
[17:04] * overclk (~quassel@117.202.101.33) Quit (Ping timeout: 480 seconds)
[17:05] * dgurtner (~dgurtner@82.199.64.68) Quit (Ping timeout: 480 seconds)
[17:06] * thomnico (~thomnico@132.217.26.109.rev.sfr.net) Quit (Ping timeout: 480 seconds)
[17:07] * vikhyat|gone (~vumrao@49.248.85.253) has joined #ceph
[17:07] * analbeard (~shw@support.memset.com) Quit (Quit: Leaving.)
[17:08] * vikhyat|gone (~vumrao@49.248.85.253) has left #ceph
[17:08] * vbellur (~vijay@71.234.224.255) Quit (Ping timeout: 480 seconds)
[17:10] * kawa2014 (~kawa@5.90.37.231) Quit (Ping timeout: 480 seconds)
[17:11] * ibravo (~ibravo@72.83.69.64) has joined #ceph
[17:13] * LDA (~lda@host217-114-156-249.pppoe.mark-itt.net) has joined #ceph
[17:13] <nils_> my daemons keep hitting suicide timeout when backfilling
[17:14] * mlovell (~mlovell@69-195-66-94.unifiedlayer.com) has joined #ceph
[17:14] * vikhyat (~vumrao@49.248.85.253) has joined #ceph
[17:15] * vikhyat is now known as vikhyat|away
[17:16] * Revo84 (~Throlkim@6AGAAAJZY.tor-irc.dnsbl.oftc.net) Quit ()
[17:19] * kawa2014 (~kawa@89.184.114.246) has joined #ceph
[17:20] * bearkitten (~bearkitte@cpe-76-172-86-115.socal.res.rr.com) Quit (Quit: WeeChat 1.4)
[17:20] * swami1 (~swami@49.32.0.112) Quit (Quit: Leaving.)
[17:21] * bearkitten (~bearkitte@cpe-76-172-86-115.socal.res.rr.com) has joined #ceph
[17:21] * caobo_ (~caobo@123.235.20.194) Quit (Quit: Leaving)
[17:22] * wjw-freebsd (~wjw@176.74.240.1) has joined #ceph
[17:24] * vata (~vata@cable-21.246.173-197.electronicbox.net) Quit (Quit: Leaving.)
[17:26] * wjw-freebsd2 (~wjw@176.74.240.1) Quit (Ping timeout: 480 seconds)
[17:29] * allaok (~allaok@machine107.orange-labs.com) has left #ceph
[17:30] <rmart04> Hello, any one know if its possible to force a refresh of the pgmap?
[17:30] * debian112 (~bcolbert@24.126.201.64) has joined #ceph
[17:35] * wyang (~wyang@114.111.166.46) Quit (Quit: This computer has gone to sleep)
[17:36] * vicente (~vicente@1-163-217-223.dynamic.hinet.net) has joined #ceph
[17:36] * EinstCrazy (~EinstCraz@58.39.19.226) Quit (Remote host closed the connection)
[17:38] * sudocat (~dibarra@192.185.1.20) has joined #ceph
[17:39] <TMM> hmm, during a rebuild I actually see the load on my osds spike to 70+
[17:39] * yguang11 (~yguang11@nat-dip27-wl-a.cfw-a-gci.corp.yahoo.com) has joined #ceph
[17:41] * debian112 (~bcolbert@24.126.201.64) Quit (Read error: Connection reset by peer)
[17:41] <TMM> Do I maybe just have too many threads per OSD?
[17:44] * DV_ (~veillard@2001:41d0:a:f29f::1) Quit (Ping timeout: 480 seconds)
[17:45] * Miouge_ (~Miouge@h-72-233.a163.priv.bahnhof.se) has joined #ceph
[17:45] * vbellur (~vijay@nat-pool-bos-t.redhat.com) has joined #ceph
[17:46] * OODavo (~Linkshot@politkovskaja.torservers.net) has joined #ceph
[17:47] <TMM> it appears my osds have around 1500 threads each
[17:47] <TMM> is that normal?
[17:47] * Skaag (~lunix@cpe-172-91-77-84.socal.res.rr.com) Quit (Quit: Leaving.)
[17:48] <rkeene> That's a bit much.
[17:48] <TMM> the ceph processes use around 300% cpu while starting a backfill, that is fine, I have cpu to spare. But I get a load of 70, and that is causing slow requests for clients, that IS a problem
[17:49] <TMM> even if all osds use 300% cpu, I'd still only have 50% of my cpus loaded
[17:49] <rmart04> can you reduce the rebuild rate by reducing the concurent backfills etc
[17:49] <m0zes> thread count is reasonable. my avg threadcount is 1410.
[17:49] <TMM> I already did that
[17:49] <rmart04> http://tracker.ceph.com/issues/15416 < ticket for issue earliar
[17:50] * b_rake (~b_rake@69-195-66-67.unifiedlayer.com) has joined #ceph
[17:51] <TMM> osd max backfills = 1
[17:51] <TMM> osd recovery max active = 1
[17:51] <TMM> osd recovery op priority = 10
[17:51] <TMM> err 1
[17:51] <TMM> still system load increases a lot, but iowait remains near 0%
[17:52] <m0zes> it might be the peering process that is causing slow requests. although I'm not sure how to speed it up.
[17:53] <rmart04> I might be way off, and I Cant see if you specified your release, but I???ve heard that some of the prioritisation isnt honored properly untill the infernalis release (unified IO queue updates) So the priority might not be helping much
[17:53] <TMM> It can't be normal that during every rebuild I'm getting user-visible artifacts of that, right?
[17:54] <rmart04> How big is your cluster?
[17:54] <TMM> 29 nodes, 8 osds per node, full ssd
[17:54] <TMM> 48 cores in each node
[17:54] <TMM> and 500gb of ram
[17:54] <TMM> connected with 2x10GB in a bond
[17:54] <rmart04> Ahh OK, I was expecting you to say 4 nodes or similar, sounds like a beast of a setup. I???m supprised your having that type of issue then!
[17:55] <TMM> not as surprised as I am
[17:55] <TMM> :P
[17:57] <rmart04> Well, at least your object count isnt currently minus many billions, look on the bright side! :)
[17:57] * shakamunyi (~shakamuny@c-67-180-191-38.hsd1.ca.comcast.net) Quit (Read error: Connection reset by peer)
[17:57] <TMM> rmart04, true, true
[17:57] <TMM> "osd op queue cut off" could that fix it? The documentation suggests that it could help if an individual OSD becomes really busy
[17:58] <TMM> what I don't understand is why any individual OSD is even getting that busy
[17:58] * debian112 (~bcolbert@24.126.201.64) has joined #ceph
[17:58] <rmart04> what are you using as journals? (though you said your %wa was low, so should not be the problem)
[17:59] <TMM> the same SSDs as the data drives
[17:59] <rmart04> iostat showing lots of activity?
[17:59] <rmart04> journals and data on same SSD?
[17:59] <rmart04> or one SSD journal for each data disk?
[17:59] <TMM> each osd has a 1TB ssd, with a 1GB journal
[17:59] <TMM> on the same ssd
[18:00] <rmart04> Ahh ok, thats kind of double writing to each disk then?
[18:00] <TMM> I don't have the issue happening right this second so I don't know the iostat, I didn't look. I haven't had time to setup some reasonable graphing yet
[18:00] <m0zes> 1GB?
[18:00] <TMM> m0zes, too small?
[18:01] <m0zes> should be 2 * min_flush_interval * max write speed of drive.
[18:01] <m0zes> min_flush_interval is 5
[18:01] <m0zes> by default
[18:01] <TMM> so how much my ssd can write in 10 seconds?
[18:01] * dkrdkr1 (~oftc-webi@72.163.220.27) has joined #ceph
[18:01] <m0zes> yep.
[18:02] <TMM> should be more like 5gb then I suppose
[18:02] <TMM> could that be causing my issues?
[18:02] * angdraug (~angdraug@c-69-181-140-42.hsd1.ca.comcast.net) has joined #ceph
[18:02] <m0zes> it would cause flushes quicker which may be triggered more often in a backfill situation...
[18:02] <TMM> and that would result in higher cpu load?
[18:03] <m0zes> I'm not sure if it would.
[18:03] <TMM> I mean, I can change it
[18:03] <TMM> it'll be annoying as fuck to do but it's not a problem
[18:04] <m0zes> it should be done for performance, surely, but I can't guarantee that this is the root problem
[18:05] <TMM> When I was benchmarking this cluster I found no difference in speed between 1gb, 5gb, and 10gb journals
[18:05] * vicente (~vicente@1-163-217-223.dynamic.hinet.net) Quit (Ping timeout: 480 seconds)
[18:05] <TMM> I figured the ssds were fast enough for it not to matter
[18:06] <m0zes> it may not matter then.
[18:07] * TomasCZ (~TomasCZ@yes.tenlab.net) has joined #ceph
[18:07] * debian112 (~bcolbert@24.126.201.64) Quit (Read error: Connection reset by peer)
[18:07] <m0zes> how fast are these ssds?
[18:07] <TMM> well, what I didn't benchmark was the performance of the cluster during backfill
[18:07] <TMM> I regret that now :P
[18:08] <m0zes> when you pull a drive (or stop an osd), and the pgs finish peering, do you still have blocked io?
[18:08] <TMM> no
[18:09] <TMM> it's not during the entire recovery I think
[18:09] <TMM> let me check the log
[18:09] * Skaag (~lunix@65.200.54.234) has joined #ceph
[18:10] <m0zes> okay. the blocked io is caused by the peering/activating stages of recovery. what version of ceph are you running?
[18:10] <TMM> yes, I saw no more slow requests once there were no more pgs in peering mode
[18:11] <TMM> the backfilling itself is not causing any issues it seems
[18:11] <TMM> I'm on hammer
[18:11] <m0zes> there was a blueprint for infernalis to get faster peering, although I'm not sure if it ever got finished. http://tracker.ceph.com/projects/ceph/wiki/Osd_-_Faster_Peering
[18:11] <TMM> 0.94.6
[18:12] <TMM> I just had a thought, does ceph create a whole lot of new threads during this process?
[18:12] <m0zes> iirc, there are threads created for every pg the osd has.
[18:12] <TMM> I have numad running on those osd machiens
[18:12] <TMM> and it was using a ton of cpu during those backfills too
[18:13] <m0zes> so so new threads would be created while peering, and they'd actively work to send/recv data from the replicas
[18:13] <TMM> and numad is shuffling them around the numa nodes
[18:13] <TMM> maybe that introduces a lot of latency?
[18:14] <m0zes> it could
[18:15] * shyu_ (~shyu@111.197.116.113) Quit (Ping timeout: 480 seconds)
[18:16] * OODavo (~Linkshot@4MJAAD16P.tor-irc.dnsbl.oftc.net) Quit ()
[18:16] * jwandborg (~hgjhgjh@exit1.ipredator.se) has joined #ceph
[18:16] * derjohn_mobi (~aj@2001:6f8:1337:0:816c:990a:d237:3b1c) Quit (Ping timeout: 480 seconds)
[18:18] <ska> When doing a 'ceph status -fjson-pretty', under health.health.health_services[0].mons, what are the 'store_stats' numbers referring to?
[18:19] * ftuesca (~ftuesca@181.170.106.36) has joined #ceph
[18:21] * Axion_Joey (~oftc-webi@static-108-47-170-18.lsanca.fios.frontiernet.net) has joined #ceph
[18:21] <Axion_Joey> Hey everyone
[18:22] <Axion_Joey> Anyone know any tricks to troubleshoot client connection problems?
[18:22] * TMM (~hp@185.5.122.2) Quit (Quit: Ex-Chat)
[18:22] * wjw-freebsd2 (~wjw@176.74.240.1) has joined #ceph
[18:25] * ngoswami (~ngoswami@121.244.87.116) Quit (Quit: Leaving)
[18:25] * debian112 (~bcolbert@24.126.201.64) has joined #ceph
[18:26] <Axion_Joey> I've got a healthy cluster, buy my client reports that it couldn't connecto to the cluster
[18:26] <Axion_Joey> and I see this in the ceph-mon logs
[18:26] <Axion_Joey> 2016-04-07 09:25:39.208618 7f8b453a8700 1 mon.proxmoxnj1@0(leader).auth v33 client did not provide supported auth type
[18:28] <etienneme> Axion_Joey: first check that the user really have rights :) "ceph auth list"
[18:28] <Axion_Joey> Thanks for your help
[18:28] <Axion_Joey> root@proxmoxnj1:/etc/pve/priv# ceph auth list installed auth entries: osd.0 key: AQCqigVXHf58LxAAxJQW1rLX7KRfZFo5lKVGvg== caps: [mon] allow profile osd caps: [osd] allow * osd.1 key: AQBKjQVXpH59LRAA4pjVdjOsm8Fnv3q4qDiyRg== caps: [mon] allow profile osd caps: [osd] allow * osd.2 key: AQBOjQVXztRGAxAAIKMgmCBlARbKGHl6V3affQ== caps: [mon] allow profile osd
[18:28] <Axion_Joey> that's wasn't very readable
[18:28] <etienneme> dont share your keys :p
[18:29] * wjw-freebsd (~wjw@176.74.240.1) Quit (Ping timeout: 480 seconds)
[18:29] <Axion_Joey> Ha ha. see I'm a noob
[18:29] <Axion_Joey> what should I look for in the auth list results?
[18:30] * dkrdkr1 (~oftc-webi@72.163.220.27) Quit (Ping timeout: 480 seconds)
[18:30] * Geph (~Geoffrey@41.77.153.99) Quit (Ping timeout: 480 seconds)
[18:31] * branto (~branto@178-253-132-82.3pp.slovanet.sk) has joined #ceph
[18:31] <etienneme> You have no user yet :)
[18:31] <Axion_Joey> uh... how do I add one?
[18:32] * jclm (~jclm@ip68-96-198-45.lv.lv.cox.net) has joined #ceph
[18:33] <etienneme> what's your version ? (ceph -v, you can give output this time ;) )
[18:33] <Axion_Joey> :)
[18:33] <Axion_Joey> ceph version 9.2.1
[18:34] * mattbenjamin1 (~mbenjamin@aa2.linuxbox.com) has joined #ceph
[18:34] <hardwire> Now that I'm seeing more action with RDMA and other backing stores, which is awesome and I'm excited!, I wonder if shared block devices and OSDs will get along so that the OSD server can operate as normal, as well as offer addressable ranges for shared storage devices (read/write) to supported clients.
[18:34] <etienneme> Axion_Joey: read this http://docs.ceph.com/docs/infernalis/rados/operations/user-management/#managing-users
[18:34] <Axion_Joey> On it. Thanks!
[18:35] * DanFoster (~Daniel@2a00:1ee0:3:1337:52b:8f9e:3178:1e1d) Quit (Quit: Leaving)
[18:35] * yguang11 (~yguang11@nat-dip27-wl-a.cfw-a-gci.corp.yahoo.com) Quit ()
[18:35] * rotbeard (~redbeard@185.32.80.238) Quit (Quit: Leaving)
[18:35] <hardwire> It would be great to be able to utilize existing SAN equipment and leverage the positive aspects of it, while letting Ceph broker all of the IO.
[18:36] * ade (~abradshaw@212.77.58.61) Quit (Ping timeout: 480 seconds)
[18:36] * haplo37 (~haplo37@199.91.185.156) has joined #ceph
[18:37] <etienneme> np :)
[18:37] <hardwire> In my case I'm upgrading the 3rd and 4th switch modules in several blade centers to 10gig Ethernet instead of fiber channel, but I still have to use the arrays (via the OSD servers which are racked elsewhere) since we're reclaiming the drives.
[18:38] <hardwire> If there were a method to have the OSDs broker access to shared storage devices over non-IP networks.. my mind would be blown.
[18:38] * vasu (~vasu@c-73-231-60-138.hsd1.ca.comcast.net) has joined #ceph
[18:38] <hardwire> And I'd just keep the fiberchannel cards in the bladeservers and use those for now.
[18:39] * wjw-freebsd2 (~wjw@176.74.240.1) Quit (Ping timeout: 480 seconds)
[18:40] <Axion_Joey> etiienneme doesn't client.admin count as a user?
[18:40] <Axion_Joey> that's might not have been pasted in my original (not smart) response, but it does exist
[18:41] <Axion_Joey> client.admin key: XXXXXXXXXXXXXXXXXXXX== caps: [mds] allow * caps: [mon] allow * caps: [osd] allow *
[18:41] * dgurtner_ (~dgurtner@82.199.64.68) Quit (Ping timeout: 480 seconds)
[18:42] * aaronb (~aaronb@2001:470:8b2d:1e1c:d267:e5ff:feec:c62c) has joined #ceph
[18:42] * aaronb (~aaronb@2001:470:8b2d:1e1c:d267:e5ff:feec:c62c) has left #ceph
[18:42] * dyasny (~dyasny@bzq-82-81-161-51.red.bezeqint.net) Quit (Ping timeout: 480 seconds)
[18:44] * aboyle (~aboyle__@2001:470:8b2d:1e1c:d267:e5ff:feec:c62c) has joined #ceph
[18:46] * jwandborg (~hgjhgjh@4MJAAD17W.tor-irc.dnsbl.oftc.net) Quit ()
[18:46] * Averad (~W|ldCraze@atlantic480.us.unmetered.com) has joined #ceph
[18:47] * Miouge_ (~Miouge@h-72-233.a163.priv.bahnhof.se) Quit (Quit: Miouge_)
[18:50] * vbellur (~vijay@nat-pool-bos-t.redhat.com) Quit (Quit: Leaving.)
[18:55] * zaitcev (~zaitcev@c-50-130-189-82.hsd1.nm.comcast.net) has joined #ceph
[18:58] * angdraug (~angdraug@c-69-181-140-42.hsd1.ca.comcast.net) Quit (Quit: Leaving)
[19:02] * farblue (~farblue@wor.namesco.net) Quit (Quit: farblue)
[19:03] * Be-El (~blinke@nat-router.computational.bio.uni-giessen.de) Quit (Quit: Leaving.)
[19:03] * Miouge_ (~Miouge@h-72-233.a163.priv.bahnhof.se) has joined #ceph
[19:07] * rmart04 (~rmart04@support.memset.com) Quit (Quit: rmart04)
[19:08] * mykola (~Mikolaj@91.225.200.223) has joined #ceph
[19:12] <etienneme> Axion_Joey: Yes, what do you want to do?
[19:13] <Axion_Joey> get the client to connect to the cluster
[19:13] <Axion_Joey> Unfortunately i created this problem myself
[19:13] <Axion_Joey> we had this working in the lab.
[19:13] * bjornar__ (~bjornar@109.247.131.38) Quit (Ping timeout: 480 seconds)
[19:13] <Axion_Joey> But I wanted to make sure I could duplicate the results
[19:14] <Axion_Joey> So I wiped everything and started from scratch
[19:14] * MrAbaddon (~MrAbaddon@193.137.26.66) Quit (Ping timeout: 480 seconds)
[19:14] <Axion_Joey> now I'm having this problem
[19:14] <etienneme> hehe :)
[19:15] <etienneme> You want to use ceph for your proxmox cluster?
[19:15] <Axion_Joey> I know it's got to be something dumb, but I can't figure it out and I've been banging my head on this for 2 days
[19:15] <Axion_Joey> yes
[19:15] <Axion_Joey> did you see my post on the proxmox forum?
[19:15] <etienneme> no
[19:15] <etienneme> But i see your hostname :p
[19:16] <Axion_Joey> :)
[19:16] <Axion_Joey> https://forum.proxmox.com/threads/ceph-authentication-error.26827/
[19:16] <Axion_Joey> that's got everything I can think of
[19:16] * Averad (~W|ldCraze@4MJAAD181.tor-irc.dnsbl.oftc.net) Quit ()
[19:20] <etienneme> http://docs.ceph.com/docs/hammer/rados/operations/user-management/#keyring-management
[19:20] <etienneme> "After you create a user (e.g., client.ringo), you must get the key and add it to a keyring on a Ceph client so that the user can access the Ceph Storage Cluster."
[19:20] <etienneme> Create your user, it will generate a keyring, use this keyring on your proxmox server.
[19:21] * gregsfortytwo (~gregsfort@206.169.83.146) has joined #ceph
[19:22] * vbellur (~vijay@71.234.224.255) has joined #ceph
[19:22] <Axion_Joey> Thanks. I have the client.admin keyring on the proxmox server now
[19:22] <Axion_Joey> That's one of the first things I did. but I can't tell if it's being used or not
[19:22] <Axion_Joey> It's in the right location and the md5 matches, so the file is correct.
[19:22] * kefu (~kefu@114.92.120.83) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[19:23] * gregsfortytwo1 (~gregsfort@transit-86-181-132-209.redhat.com) Quit (Ping timeout: 480 seconds)
[19:23] * xarses (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[19:24] <vasu> Axion_Joey: check this doc http://docs.ceph.com/docs/hammer/rados/operations/user-management/
[19:25] * jproulx (~jon@kvas.csail.mit.edu) has left #ceph
[19:27] <Axion_Joey> Reading it vasu
[19:27] <Axion_Joey> I'll create and try a user other than admin
[19:29] * vata (~vata@cable-21.246.173-197.electronicbox.net) has joined #ceph
[19:35] * kefu (~kefu@114.92.120.83) has joined #ceph
[19:36] * igi (~igi@14.146.39.42) has joined #ceph
[19:38] <igi> hello, guys, i just wondered what unit the latency is stated in rados bench.
[19:39] <igi> seconds or millisecond?
[19:40] <gregsfortytwo> seconds
[19:42] <igi> thanks
[19:44] * bniver (~bniver@71-9-144-29.static.oxfr.ma.charter.com) Quit (Remote host closed the connection)
[19:45] <igi> someone had added a ticket to the tracker to request add unit for latency, but without any response about it.
[19:45] * ftuesca (~ftuesca@181.170.106.36) Quit (Quit: Leaving)
[19:45] <igi> it will be more clearly that state the units in the output.
[19:46] * Zombiekiller (~andrew_m@tor-relay.zwiebeltoralf.de) has joined #ceph
[19:46] * jordanP (~jordan@204.13-14-84.ripe.coltfrance.com) Quit (Quit: Leaving)
[19:47] * wwdillingham (~LobsterRo@65.112.8.195) Quit (Quit: wwdillingham)
[19:48] * overclk_ (~quassel@117.202.96.251) Quit (Remote host closed the connection)
[19:52] * rmart04 (~rmart04@support.memset.com) has joined #ceph
[19:52] * gregsfortytwo1 (~gregsfort@transit-86-181-132-209.redhat.com) has joined #ceph
[19:56] * xarses (~xarses@64.124.158.100) has joined #ceph
[20:01] * bjornar__ (~bjornar@ti0099a430-1561.bb.online.no) has joined #ceph
[20:04] * BrianA1 (~BrianA@c-50-168-46-112.hsd1.ca.comcast.net) has joined #ceph
[20:13] * haplo37 (~haplo37@199.91.185.156) Quit (Ping timeout: 480 seconds)
[20:16] <LDA> hi! Hoto correct shutdown whole cluster ? set flags noout,nodown,nobackfill,norebalance,norecover? befor shutdown.
[20:16] * Zombiekiller (~andrew_m@6AGAAAKC0.tor-irc.dnsbl.oftc.net) Quit ()
[20:16] * CoZmicShReddeR (~Lunk2@65.19.167.130) has joined #ceph
[20:22] * wwdillingham (~LobsterRo@140.247.242.44) has joined #ceph
[20:23] * i_m (~ivan.miro@deibp9eh1--blueice3n0.emea.ibm.com) Quit (Quit: Leaving.)
[20:24] * BrianA1 (~BrianA@c-50-168-46-112.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[20:27] * nils_ (~nils_@doomstreet.collins.kg) Quit (Quit: This computer has gone to sleep)
[20:30] * vikhyat|away (~vumrao@49.248.85.253) Quit (Quit: Leaving)
[20:31] * gregsfortytwo (~gregsfort@206.169.83.146) Quit (Quit: Textual IRC Client: www.textualapp.com)
[20:34] * kefu (~kefu@114.92.120.83) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[20:34] * bniver (~bniver@71-9-144-29.static.oxfr.ma.charter.com) has joined #ceph
[20:35] * MrAbaddon (~MrAbaddon@a89-155-99-93.cpe.netcabo.pt) has joined #ceph
[20:35] * MrAbaddon (~MrAbaddon@a89-155-99-93.cpe.netcabo.pt) Quit ()
[20:36] * MrAbaddon (~MrAbaddon@a89-155-99-93.cpe.netcabo.pt) has joined #ceph
[20:39] <rmart04> rmart04: Just incase anyone might have a few mins to spare ill just post this here??? http://tracker.ceph.com/issues/15416
[20:39] <rmart04> rmart04: Looking for a way to reset/rebuild the pgmap
[20:39] <rmart04> rmart04: if thats even a thing.
[20:41] * mhack (~mhack@nat-pool-bos-t.redhat.com) Quit (Ping timeout: 480 seconds)
[20:44] * angdraug (~angdraug@64.124.158.100) has joined #ceph
[20:46] * CoZmicShReddeR (~Lunk2@4MJAAD2BX.tor-irc.dnsbl.oftc.net) Quit ()
[20:49] * igi (~igi@14.146.39.42) Quit (Quit: Leaving...)
[20:51] * bniver (~bniver@71-9-144-29.static.oxfr.ma.charter.com) Quit (Remote host closed the connection)
[20:52] * mhack (~mhack@nat-pool-bos-u.redhat.com) has joined #ceph
[20:55] * bniver (~bniver@71-9-144-29.static.oxfr.ma.charter.com) has joined #ceph
[20:56] * TMM (~hp@178-84-46-106.dynamic.upc.nl) has joined #ceph
[20:59] * markednmbr1 (~Diego@cpc1-lewi13-2-0-cust267.2-4.cable.virginm.net) Quit (Quit: Leaving)
[21:00] <TMM> m0zes, do you think that my backfill max scan should be lowered maybe? Would that decrease load under backfill? Or am I completely misunderstanding what that parameter does
[21:00] * kawa2014 (~kawa@89.184.114.246) Quit (Quit: Leaving)
[21:01] * debian112 (~bcolbert@24.126.201.64) Quit (Read error: Connection reset by peer)
[21:02] * wjw-freebsd (~wjw@smtp.digiware.nl) has joined #ceph
[21:02] * branto (~branto@178-253-132-82.3pp.slovanet.sk) Quit (Quit: Leaving.)
[21:04] <TMM> I'm beginning to wonder if I'd maybe just be better off with cfq so I can have io priorities
[21:04] <TMM> I'm currently on noop for my ssd osdd
[21:04] * Miouge_ (~Miouge@h-72-233.a163.priv.bahnhof.se) Quit (Quit: Miouge_)
[21:06] * swami1 (~swami@27.7.161.87) has joined #ceph
[21:07] * rendar (~I@95.234.180.115) Quit (Ping timeout: 480 seconds)
[21:07] * analbeard (~shw@host86-140-202-165.range86-140.btcentralplus.com) has joined #ceph
[21:07] <analbeard> evening all
[21:08] <analbeard> can anyone suggest any ideas regarding this issue I raised? http://tracker.ceph.com/issues/15416
[21:08] * rwheeler (~rwheeler@nat-pool-bos-t.redhat.com) Quit (Quit: Leaving)
[21:08] <analbeard> some odd output in ceph df and it's causing issues with openstack
[21:10] * rendar (~I@95.234.180.115) has joined #ceph
[21:12] * BrianA1 (~BrianA@c-50-168-46-112.hsd1.ca.comcast.net) has joined #ceph
[21:13] * Discovery (~Discovery@178.239.49.68) has joined #ceph
[21:15] <ska> Can I bump up the number of PG's in my OSDs?
[21:16] * mykola (~Mikolaj@91.225.200.223) Quit (Remote host closed the connection)
[21:16] <analbeard> ska: you can, however there are some guidelines you should probably look at rather than just bumping it
[21:16] * Jones (~Curt`@7V7AAD3BM.tor-irc.dnsbl.oftc.net) has joined #ceph
[21:16] <analbeard> ska: http://ceph.com/pgcalc/
[21:20] * debian112 (~bcolbert@24.126.201.64) has joined #ceph
[21:21] * mhackett (~mhack@nat-pool-bos-t.redhat.com) has joined #ceph
[21:23] * dgurtner (~dgurtner@82.199.64.68) has joined #ceph
[21:26] * BrianA1 (~BrianA@c-50-168-46-112.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[21:27] * mhack (~mhack@nat-pool-bos-u.redhat.com) Quit (Ping timeout: 480 seconds)
[21:46] * Jones (~Curt`@7V7AAD3BM.tor-irc.dnsbl.oftc.net) Quit ()
[21:46] * Tenk (~Nephyrin@195.22.126.119) has joined #ceph
[21:51] <ska> analbeard: Sure.. I realize its dangerous, but this is a testing cluster.
[21:51] <ska> We're actually trying to drive it mad.
[21:52] * ade (~abradshaw@80-72-52-180.dynamic.modem.fl1.li) has joined #ceph
[21:52] * mhackett is now known as mhack
[21:53] * allaok (~allaok@ARennes-658-1-126-166.w83-199.abo.wanadoo.fr) has joined #ceph
[22:00] * evelu (~erwan@46.231.131.178) Quit (Ping timeout: 480 seconds)
[22:05] <brians__> anyone running samsung 850 pros as storage (not journals) ?
[22:06] * LDA (~lda@host217-114-156-249.pppoe.mark-itt.net) Quit (Quit: LDA)
[22:08] * descenti1n (~descentio@c-50-152-79-210.hsd1.pa.comcast.net) Quit (Remote host closed the connection)
[22:16] * Miouge_ (~Miouge@h-72-233.a163.priv.bahnhof.se) has joined #ceph
[22:16] * analbeard (~shw@host86-140-202-165.range86-140.btcentralplus.com) Quit (Ping timeout: 480 seconds)
[22:16] * Tenk (~Nephyrin@4MJAAD2E3.tor-irc.dnsbl.oftc.net) Quit ()
[22:16] * SurfMaths (~andrew_m@92.222.103.234) has joined #ceph
[22:23] * derjohn_mobi (~aj@x4db1216c.dyn.telefonica.de) has joined #ceph
[22:25] * rwheeler (~rwheeler@pool-173-48-195-215.bstnma.fios.verizon.net) has joined #ceph
[22:28] * georgem (~Adium@206.108.127.16) Quit (Ping timeout: 480 seconds)
[22:34] * Discovery (~Discovery@178.239.49.68) Quit (Read error: Connection reset by peer)
[22:38] * b_rake (~b_rake@69-195-66-67.unifiedlayer.com) Quit (Remote host closed the connection)
[22:39] * whatevsz (~quassel@ip5f5adec4.dynamic.kabel-deutschland.de) Quit (Remote host closed the connection)
[22:40] * whatevsz (~quassel@ip5f5adec4.dynamic.kabel-deutschland.de) has joined #ceph
[22:40] * whatevsz (~quassel@ip5f5adec4.dynamic.kabel-deutschland.de) Quit (Remote host closed the connection)
[22:46] * debian112 (~bcolbert@24.126.201.64) Quit (Read error: Connection reset by peer)
[22:46] * SurfMaths (~andrew_m@06SAAA2MJ.tor-irc.dnsbl.oftc.net) Quit ()
[22:46] * b_rake (~b_rake@69-195-66-67.unifiedlayer.com) has joined #ceph
[22:46] * Nephyrin (~Ian2128@torlesnet2.relay.coldhak.com) has joined #ceph
[22:49] * dgurtner_ (~dgurtner@82.199.64.68) has joined #ceph
[22:49] * analbeard (~shw@host86-140-202-165.range86-140.btcentralplus.com) has joined #ceph
[22:51] * dgurtner (~dgurtner@82.199.64.68) Quit (Ping timeout: 480 seconds)
[22:51] * whatevsz (~quassel@b9168eb2.cgn.dg-w.de) has joined #ceph
[22:52] * T1 (~the_one@87.104.212.66) Quit (Quit: Where did the client go?)
[22:52] * whatevsz (~quassel@b9168eb2.cgn.dg-w.de) Quit ()
[22:53] * whatevsz (~quassel@b9168eb2.cgn.dg-w.de) has joined #ceph
[22:54] * b_rake (~b_rake@69-195-66-67.unifiedlayer.com) Quit (Read error: Connection reset by peer)
[22:56] * T1 (~the_one@87.104.212.66) has joined #ceph
[22:58] * wwdillingham_ (~LobsterRo@65.112.8.195) has joined #ceph
[22:58] * wwdillingham_ (~LobsterRo@65.112.8.195) Quit ()
[22:59] * bniver (~bniver@71-9-144-29.static.oxfr.ma.charter.com) Quit (Remote host closed the connection)
[23:01] * georgem (~Adium@24.114.65.209) has joined #ceph
[23:01] * georgem (~Adium@24.114.65.209) Quit ()
[23:01] * georgem (~Adium@206.108.127.16) has joined #ceph
[23:03] * rendar (~I@95.234.180.115) Quit (Quit: std::lower_bound + std::less_equal *works* with a vector without duplicates!)
[23:04] * debian112 (~bcolbert@24.126.201.64) has joined #ceph
[23:05] * wwdillingham (~LobsterRo@140.247.242.44) Quit (Ping timeout: 480 seconds)
[23:06] * ade (~abradshaw@80-72-52-180.dynamic.modem.fl1.li) Quit (Ping timeout: 480 seconds)
[23:10] * bene3 (~bene@2601:18c:8501:25e4:ea2a:eaff:fe08:3c7a) Quit (Quit: Konversation terminated!)
[23:10] * Miouge_ (~Miouge@h-72-233.a163.priv.bahnhof.se) Quit (Quit: Miouge_)
[23:14] * debian112 (~bcolbert@24.126.201.64) Quit (Read error: Connection reset by peer)
[23:14] * whatevsz (~quassel@b9168eb2.cgn.dg-w.de) Quit (Remote host closed the connection)
[23:15] * whatevsz (~quassel@b9168eb2.cgn.dg-w.de) has joined #ceph
[23:15] * danieagle (~Daniel@187.35.207.195) has joined #ceph
[23:16] * whatevsz (~quassel@b9168eb2.cgn.dg-w.de) Quit (Remote host closed the connection)
[23:16] * whatevsz (~quassel@b9168eb2.cgn.dg-w.de) has joined #ceph
[23:16] * Nephyrin (~Ian2128@06SAAA2NJ.tor-irc.dnsbl.oftc.net) Quit ()
[23:16] * _28_ria (~kvirc@opfr028.ru) has joined #ceph
[23:17] * shakamunyi (~shakamuny@c-67-180-191-38.hsd1.ca.comcast.net) has joined #ceph
[23:20] * boredatwork (~overonthe@199.68.193.54) has joined #ceph
[23:21] * swami1 (~swami@27.7.161.87) Quit (Ping timeout: 480 seconds)
[23:24] * whatevsz (~quassel@b9168eb2.cgn.dg-w.de) Quit (Remote host closed the connection)
[23:24] * whatevsz (~quassel@b9168eb2.cgn.dg-w.de) has joined #ceph
[23:27] * whatevsz (~quassel@b9168eb2.cgn.dg-w.de) Quit (Remote host closed the connection)
[23:27] * whatevsz (~quassel@b9168eb2.cgn.dg-w.de) has joined #ceph
[23:27] * dynamicudpate (~overonthe@199.68.193.54) Quit (Ping timeout: 480 seconds)
[23:27] * rburkholder (~overonthe@199.68.193.54) Quit (Ping timeout: 480 seconds)
[23:27] * shakamunyi (~shakamuny@c-67-180-191-38.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[23:27] * rburkholder (~overonthe@199.68.193.54) has joined #ceph
[23:31] * dgurtner (~dgurtner@217.149.140.193) has joined #ceph
[23:33] * dgurtner_ (~dgurtner@82.199.64.68) Quit (Ping timeout: 480 seconds)
[23:33] * debian112 (~bcolbert@24.126.201.64) has joined #ceph
[23:34] * whatevsz (~quassel@b9168eb2.cgn.dg-w.de) Quit (Remote host closed the connection)
[23:35] * whatevsz (~quassel@b9168eb2.cgn.dg-w.de) has joined #ceph
[23:37] * mhack (~mhack@nat-pool-bos-t.redhat.com) Quit (Remote host closed the connection)
[23:40] * analbeard (~shw@host86-140-202-165.range86-140.btcentralplus.com) Quit (Quit: Leaving.)
[23:42] * georgem (~Adium@206.108.127.16) Quit (Read error: Connection reset by peer)
[23:45] * Rickus (~Rickus@office.protected.ca) has joined #ceph
[23:46] <bdeetz> Would anybody be interested/willing to comment on this post on the mailing list? http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-April/008775.html
[23:49] <m0zes> I would call that setup pretty dense. ceph seems to shine much more with more smaller nodes.
[23:51] <m0zes> other than that, it looks nice. I've put 10 osds on a single p3700 for journaling, it held up well. I don't know if it would stand up as well to 18 osds.
[23:52] <m0zes> you've more than maxed out the available bandwidth to the ssd at that point.
[23:52] <bdeetz> So would you consider 24 OSDs more in the sweet spot? I really don't want to move down to 12 just because of the cost of the chassis.
[23:53] <bdeetz> Maybe just add an additional SSD if I have a pcie slot available?
[23:54] <bdeetz> Also, any opinions on interconnect? I have experience with IB, but I'm not sure it's a good fit here. 40g is enticing.
[23:55] <bdeetz> seeing as RDMA isn't built-in yet
[23:55] <m0zes> 40G is nice. that is what I'm using.
[23:55] <m0zes> 24 osds could work. how many hbas are driving the disks?
[23:56] <lurbs> Because you need 40G, or because of latency?
[23:56] <ingard> bdeetz: thats roughly our setup atm
[23:56] <ingard> we're running with erasurecoded pool tho
[23:56] <m0zes> latency. although rebuilds have pushed bandwidth past 10Gb.
[23:56] <ingard> object store only
[23:57] <ingard> using consumer drives hgst 6T x30 and 6 x journal drvies
[23:57] <ingard> rebalance is really shit for us tho
[23:58] <ingard> drives max out at 200 iops/10M read/s
[23:58] <ingard> apart from that it was inktank that helped us build it in the first place
[23:59] <bdeetz> are any of you using cephfs at the moment?
[23:59] * m0zes since hammer

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.