#ceph IRC Log

Index

IRC Log for 2016-07-25

Timestamps are in GMT/BST.

[0:06] * IvanJobs (~ivanjobs@103.50.11.146) Quit (Ping timeout: 480 seconds)
[0:17] * PappI (~Averad@178-175-128-50.static.host) has joined #ceph
[0:24] * sudocat (~dibarra@104-188-116-197.lightspeed.hstntx.sbcglobal.net) Quit (Ping timeout: 480 seconds)
[0:29] * theTrav_ (~theTrav@CPE-124-188-218-238.sfcz1.cht.bigpond.net.au) Quit (Remote host closed the connection)
[0:30] * rendar (~I@host194-41-dynamic.48-82-r.retail.telecomitalia.it) Quit (Ping timeout: 480 seconds)
[0:36] * Nacer (~Nacer@37.160.0.229) has joined #ceph
[0:44] * dnunez (~dnunez@c-73-38-0-185.hsd1.ma.comcast.net) Quit (Quit: Leaving)
[0:46] * PappI (~Averad@26XAAAKSE.tor-irc.dnsbl.oftc.net) Quit ()
[0:47] * kuku (~kuku@119.93.91.136) has joined #ceph
[0:54] * Nacer (~Nacer@37.160.0.229) Quit (Ping timeout: 480 seconds)
[1:00] * danieagle (~Daniel@201-69-183-120.dial-up.telesp.net.br) Quit (Quit: Obrigado por Tudo! :-) inte+ :-))
[1:04] * sebastian-w (~quassel@212.218.8.138) Quit (Remote host closed the connection)
[1:04] * Jaska (~JamesHarr@tsn109-201-154-142.dyn.nltelcom.net) has joined #ceph
[1:04] * sebastian-w (~quassel@212.218.8.138) has joined #ceph
[1:14] * praveen (~praveen@122.172.140.150) Quit (Remote host closed the connection)
[1:19] * vbellur (~vijay@71.234.224.255) has joined #ceph
[1:29] * theTrav (~theTrav@203.35.9.142) has joined #ceph
[1:34] * Jaska (~JamesHarr@61TAAAUFS.tor-irc.dnsbl.oftc.net) Quit ()
[1:42] * theTrav_ (~theTrav@203.35.9.142) has joined #ceph
[1:43] * IvanJobs (~ivanjobs@103.50.11.146) has joined #ceph
[1:48] * theTrav (~theTrav@203.35.9.142) Quit (Read error: Connection timed out)
[1:48] * maku1 (~Kurimus@178-175-128-50.static.host) has joined #ceph
[2:03] * oms101 (~oms101@p20030057EA58C000C6D987FFFE4339A1.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[2:11] * oms101 (~oms101@p20030057EA042A00C6D987FFFE4339A1.dip0.t-ipconnect.de) has joined #ceph
[2:18] * maku1 (~Kurimus@26XAAAKT1.tor-irc.dnsbl.oftc.net) Quit ()
[2:45] * sudocat (~dibarra@104-188-116-197.lightspeed.hstntx.sbcglobal.net) has joined #ceph
[2:52] * KindOne_ (kindone@198.14.197.7) has joined #ceph
[2:53] * sudocat (~dibarra@104-188-116-197.lightspeed.hstntx.sbcglobal.net) Quit (Ping timeout: 480 seconds)
[2:56] * yanzheng (~zhyan@118.116.114.34) has joined #ceph
[2:58] * KindOne (kindone@0001a7db.user.oftc.net) Quit (Ping timeout: 480 seconds)
[2:58] * KindOne_ is now known as KindOne
[3:04] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) Quit (Quit: neurodrone_)
[3:04] * sudocat (~dibarra@192.185.1.20) has joined #ceph
[3:04] * cronburg (~cronburg@50.245.61.156) Quit (Ping timeout: 480 seconds)
[3:09] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) has joined #ceph
[3:10] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) Quit ()
[3:12] * Hejt (~Zeis@46.101.169.151) has joined #ceph
[3:16] * kefu (~kefu@183.193.112.90) has joined #ceph
[3:21] * kefu_ (~kefu@114.92.96.253) has joined #ceph
[3:28] * kefu (~kefu@183.193.112.90) Quit (Ping timeout: 480 seconds)
[3:30] * kefu_ (~kefu@114.92.96.253) Quit (Max SendQ exceeded)
[3:30] * kefu (~kefu@114.92.96.253) has joined #ceph
[3:35] * derjohn_mobi (~aj@x4db24d74.dyn.telefonica.de) has joined #ceph
[3:36] * cronburg (~cronburg@209-6-121-249.c3-0.arl-ubr1.sbo-arl.ma.cable.rcn.com) has joined #ceph
[3:40] * sebastian-w (~quassel@212.218.8.138) Quit (Read error: Connection reset by peer)
[3:40] * sebastian-w (~quassel@212.218.8.139) has joined #ceph
[3:42] * Hejt (~Zeis@5AEAAAJU8.tor-irc.dnsbl.oftc.net) Quit ()
[3:42] * aj__ (~aj@x4db2546b.dyn.telefonica.de) Quit (Ping timeout: 480 seconds)
[3:47] * dlan (~dennis@116.228.88.131) Quit (Remote host closed the connection)
[3:50] * Jeffrey4l (~Jeffrey@121.16.247.68) has joined #ceph
[3:51] * tallest_red (~Rens2Sea@tor-exit.squirrel.theremailer.net) has joined #ceph
[4:06] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) has joined #ceph
[4:09] * EthanL (~lamberet@cce02cs4040-fa12-z.ams.hpecore.net) has joined #ceph
[4:15] * theTrav_ (~theTrav@203.35.9.142) Quit (Quit: Leaving...)
[4:15] * kuku (~kuku@119.93.91.136) Quit (Remote host closed the connection)
[4:19] * tserong (~tserong@203-214-92-220.dyn.iinet.net.au) has joined #ceph
[4:21] * tallest_red (~Rens2Sea@26XAAAKWU.tor-irc.dnsbl.oftc.net) Quit ()
[4:23] * EthanL (~lamberet@cce02cs4040-fa12-z.ams.hpecore.net) Quit (Ping timeout: 480 seconds)
[4:25] * galaxyAbstractor (~AotC@31-168-172-144.telavivwifi.com) has joined #ceph
[4:38] * KindOne_ (kindone@198.14.195.54) has joined #ceph
[4:44] * KindOne (kindone@0001a7db.user.oftc.net) Quit (Ping timeout: 480 seconds)
[4:44] * KindOne_ is now known as KindOne
[4:46] * vicente (~~vicente@125-227-238-55.HINET-IP.hinet.net) has joined #ceph
[4:51] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) Quit (Quit: neurodrone_)
[4:52] * kuku (~kuku@119.93.91.136) has joined #ceph
[4:55] * galaxyAbstractor (~AotC@31-168-172-144.telavivwifi.com) Quit ()
[5:03] * mdxi (~mdxi@li925-141.members.linode.com) Quit (Quit: leaving)
[5:14] * EinstCrazy (~EinstCraz@180.166.44.202) has joined #ceph
[5:14] * rdas (~rdas@122.168.241.13) has joined #ceph
[5:16] * John341 (~ceph@118.200.221.105) Quit (Remote host closed the connection)
[5:18] * Sirrush (~Chaos_Lla@tor2r.ins.tor.net.eu.org) has joined #ceph
[5:25] * EinstCrazy (~EinstCraz@180.166.44.202) Quit (Remote host closed the connection)
[5:26] * sudocat (~dibarra@192.185.1.20) Quit (Ping timeout: 480 seconds)
[5:30] * sudocat (~dibarra@104-188-116-197.lightspeed.hstntx.sbcglobal.net) has joined #ceph
[5:41] * sudocat (~dibarra@104-188-116-197.lightspeed.hstntx.sbcglobal.net) Quit (Ping timeout: 480 seconds)
[5:41] * vimal (~vikumar@114.143.165.70) has joined #ceph
[5:42] * Sirrush (~Chaos_Lla@61TAAAUK9.tor-irc.dnsbl.oftc.net) Quit (Ping timeout: 480 seconds)
[5:53] * sudocat (~dibarra@192.185.1.20) has joined #ceph
[5:57] * Vacuum_ (~Vacuum@88.130.221.47) has joined #ceph
[6:04] * r0lland (~r0lland@121.244.155.8) has joined #ceph
[6:04] * Vacuum__ (~Vacuum@88.130.209.75) Quit (Ping timeout: 480 seconds)
[6:07] * sudocat1 (~dibarra@104-188-116-197.lightspeed.hstntx.sbcglobal.net) has joined #ceph
[6:08] * sudocat (~dibarra@192.185.1.20) Quit (Quit: Leaving.)
[6:09] * cyphase (~cyphase@000134f2.user.oftc.net) Quit (Ping timeout: 480 seconds)
[6:11] * cyphase (~cyphase@000134f2.user.oftc.net) has joined #ceph
[6:15] * sudocat1 (~dibarra@104-188-116-197.lightspeed.hstntx.sbcglobal.net) Quit (Ping timeout: 480 seconds)
[6:21] * r0lland (~r0lland@121.244.155.8) Quit (Ping timeout: 480 seconds)
[6:23] * derjohn_mobi (~aj@x4db24d74.dyn.telefonica.de) Quit (Remote host closed the connection)
[6:27] * sudocat (~dibarra@192.185.1.20) has joined #ceph
[6:38] * vimal (~vikumar@114.143.165.70) Quit (Quit: Leaving)
[6:55] * Mattress (~Swompie`@108.61.123.78) has joined #ceph
[6:58] * vimal (~vikumar@121.244.87.116) has joined #ceph
[6:59] * cholcombe (~chris@50-206-35-84.infopact.nl) Quit (Ping timeout: 480 seconds)
[6:59] * cholcombe (~chris@50-206-35-84.infopact.nl) has joined #ceph
[7:06] * Nacer (~Nacer@93-33-160-63.ip45.fastwebnet.it) has joined #ceph
[7:07] * Nacer (~Nacer@93-33-160-63.ip45.fastwebnet.it) Quit (Remote host closed the connection)
[7:11] * vikhyat (~vumrao@121.244.87.116) has joined #ceph
[7:14] * TomasCZ (~TomasCZ@yes.tenlab.net) Quit (Quit: Leaving)
[7:25] * Mattress (~Swompie`@9YSAAAU79.tor-irc.dnsbl.oftc.net) Quit ()
[7:26] * jcsp (~jspray@82-71-16-249.dsl.in-addr.zen.co.uk) Quit (Ping timeout: 480 seconds)
[7:36] * jcsp (~jspray@82-71-16-249.dsl.in-addr.zen.co.uk) has joined #ceph
[7:36] * derjohn_mob (~aj@88.128.80.165) has joined #ceph
[7:41] * coreping (~Michael_G@n1.coreping.org) has joined #ceph
[7:47] * EinstCrazy (~EinstCraz@117.136.8.232) has joined #ceph
[7:59] * cholcombe (~chris@50-206-35-84.infopact.nl) Quit (Ping timeout: 480 seconds)
[8:04] * EinstCrazy (~EinstCraz@117.136.8.232) Quit (Ping timeout: 480 seconds)
[8:05] * dgurtner (~dgurtner@84-73-130-19.dclient.hispeed.ch) has joined #ceph
[8:07] * penguinRaider (~KiKo@146.185.31.226) Quit (Ping timeout: 480 seconds)
[8:08] * dec (~dec@45.96.198.104.bc.googleusercontent.com) has joined #ceph
[8:14] * sudocat (~dibarra@192.185.1.20) Quit (Ping timeout: 480 seconds)
[8:16] * yanzheng (~zhyan@118.116.114.34) Quit (Ping timeout: 480 seconds)
[8:19] * penguinRaider (~KiKo@146.185.31.226) has joined #ceph
[8:19] * epicguy (~epicguy@41.164.8.42) has joined #ceph
[8:19] * karnan (~karnan@121.244.87.117) has joined #ceph
[8:19] * jan (~jan@p20030084AF27B0005EC5D4FFFEBB68A4.dip0.t-ipconnect.de) has joined #ceph
[8:20] * jan is now known as Guest4175
[8:21] * Guest4175 (~jan@p20030084AF27B0005EC5D4FFFEBB68A4.dip0.t-ipconnect.de) Quit ()
[8:21] * jfaj (~jan@p20030084AF27B0005EC5D4FFFEBB68A4.dip0.t-ipconnect.de) has joined #ceph
[8:28] * yanzheng (~zhyan@118.116.114.34) has joined #ceph
[8:31] * badone (~badone@66.187.239.16) has joined #ceph
[8:31] * kawa2014 (~kawa@89.184.114.246) has joined #ceph
[8:36] * Miouge (~Miouge@109.128.94.173) has joined #ceph
[8:40] * praveen (~praveen@122.172.140.150) has joined #ceph
[8:43] * nardial (~ls@dslb-088-074-120-161.088.074.pools.vodafone-ip.de) has joined #ceph
[8:45] * briner (~briner@2001:620:600:1000:5d26:8eaa:97f0:8115) Quit (Quit: briner)
[8:47] * derjohn_mob (~aj@88.128.80.165) Quit (Ping timeout: 480 seconds)
[8:48] * briner (~briner@129.194.16.54) has joined #ceph
[9:01] * evelu (~erwan@46.231.131.178) has joined #ceph
[9:02] * rdas (~rdas@122.168.241.13) Quit (Ping timeout: 480 seconds)
[9:06] * krypto (~krypto@G68-90-105-253.sbcis.sbc.com) has joined #ceph
[9:14] * rdas (~rdas@122.168.253.92) has joined #ceph
[9:24] * Hemanth (~hkumar_@121.244.87.117) has joined #ceph
[9:27] * Jeffrey4l_ (~Jeffrey@121.16.111.97) has joined #ceph
[9:30] * Jeffrey4l (~Jeffrey@121.16.247.68) Quit (Ping timeout: 480 seconds)
[9:38] * b0e (~aledermue@213.95.25.82) has joined #ceph
[9:39] * Nicho1as (~oftc-webi@218.147.181.208) has joined #ceph
[9:40] * praveen (~praveen@122.172.140.150) Quit (Remote host closed the connection)
[9:42] * thomnico (~thomnico@2a01:e35:8b41:120:447b:2bf:dd19:182c) has joined #ceph
[9:44] * nardial (~ls@dslb-088-074-120-161.088.074.pools.vodafone-ip.de) Quit (Quit: Leaving)
[9:45] * fsimonce (~simon@host99-64-dynamic.27-79-r.retail.telecomitalia.it) has joined #ceph
[9:48] * derjohn_mob (~aj@fw.gkh-setu.de) has joined #ceph
[9:49] * Nicho1as (~oftc-webi@00022427.user.oftc.net) Quit ()
[9:50] * rakeshgm (~rakesh@121.244.87.117) has joined #ceph
[9:52] * georgem (~Adium@85.204.4.209) has joined #ceph
[9:55] * `Jin (~AGaW@46.166.190.178) has joined #ceph
[9:59] * TMM (~hp@178-84-46-106.dynamic.upc.nl) Quit (Quit: Ex-Chat)
[10:01] * allaok (~allaok@machine107.orange-labs.com) has joined #ceph
[10:02] * kuku (~kuku@119.93.91.136) Quit (Remote host closed the connection)
[10:08] * derjohn_mob (~aj@fw.gkh-setu.de) Quit (Remote host closed the connection)
[10:12] * EinstCrazy (~EinstCraz@117.136.8.232) has joined #ceph
[10:14] * gmoro (~guilherme@193.120.208.221) Quit (Remote host closed the connection)
[10:15] * hybrid512 (~walid@195.200.189.206) has joined #ceph
[10:18] * derjohn_mob (~aj@fw.gkh-setu.de) has joined #ceph
[10:19] * gmoro (~guilherme@193.120.208.221) has joined #ceph
[10:20] * praveen (~praveen@121.244.155.11) has joined #ceph
[10:20] * derjohn_mob (~aj@fw.gkh-setu.de) Quit (Remote host closed the connection)
[10:21] * georgem (~Adium@85.204.4.209) Quit (Quit: Leaving.)
[10:22] * Nicholas (~nicho1as@218.147.181.208) has joined #ceph
[10:23] * Nicholas (~nicho1as@218.147.181.208) Quit ()
[10:25] * `Jin (~AGaW@46.166.190.178) Quit ()
[10:27] * Nicholas (~nicho1as@218.147.181.208) has joined #ceph
[10:27] * Nicholas (~nicho1as@218.147.181.208) Quit ()
[10:27] * Nicho1as (~nicho1as@00022427.user.oftc.net) has joined #ceph
[10:29] * mattch (~mattch@w5430.see.ed.ac.uk) has joined #ceph
[10:30] * EinstCra_ (~EinstCraz@117.136.8.231) has joined #ceph
[10:30] <MrBy> It looks like that I have a garbage collection issue, I get "RGWGC::process() failed to acquire lock on gc.11" someone has had something similar and knows how to solve it?
[10:30] * derjohn_mob (~aj@fw.gkh-setu.de) has joined #ceph
[10:30] * EinstCrazy (~EinstCraz@117.136.8.232) Quit (Read error: No route to host)
[10:32] * branto (~branto@178-253-133-229.3pp.slovanet.sk) has joined #ceph
[10:41] * EinstCra_ (~EinstCraz@117.136.8.231) Quit (Ping timeout: 480 seconds)
[10:46] * EinstCrazy (~EinstCraz@117.136.8.231) has joined #ceph
[10:47] * TMM (~hp@185.5.121.201) has joined #ceph
[10:48] * thomnico (~thomnico@2a01:e35:8b41:120:447b:2bf:dd19:182c) Quit (Quit: Ex-Chat)
[10:57] * t4nk587 (~oftc-webi@115.119.152.66.static-hyderabad.vsnl.net.in) Quit (Ping timeout: 480 seconds)
[10:59] * Nicho1as (~nicho1as@00022427.user.oftc.net) Quit (Quit: A man from the Far East; using WeeChat 1.5)
[11:00] * rendar (~I@host150-177-dynamic.10-87-r.retail.telecomitalia.it) has joined #ceph
[11:02] * EinstCrazy (~EinstCraz@117.136.8.231) Quit (Read error: Connection timed out)
[11:03] * owasserm (~owasserm@2001:984:d3f7:1:5ec5:d4ff:fee0:f6dc) has joined #ceph
[11:06] * rotbeard (~redbeard@185.32.80.238) has joined #ceph
[11:08] * epicguy (~epicguy@41.164.8.42) Quit (Quit: Leaving)
[11:11] * t4nk643 (~oftc-webi@mintzer.imp.fu-berlin.de) Quit (Quit: Page closed)
[11:13] * EinstCrazy (~EinstCraz@117.136.8.231) has joined #ceph
[11:15] * b0e (~aledermue@213.95.25.82) Quit (Ping timeout: 480 seconds)
[11:28] * b0e (~aledermue@213.95.25.82) has joined #ceph
[11:29] * nikbor (~n.borisov@admins.1h.com) has joined #ceph
[11:29] <nikbor> hello I setup a small test cluster
[11:30] <nikbor> and running fio on an rbd device from one particular server shows very bad performance e.g. ~200IOPS and around 1mb writes per second, and the servers are connected via IPoIB (which achieves around 20gbit using netperf)
[11:30] <nikbor> what can I do to debug this ?
[11:31] <nikbor> the servers are idle
[11:32] <nikbor> the running kernels are recent 4.4 and 4.5
[11:32] <nikbor> and the version is 0.94.7
[11:38] <IcePic> nikbor: how does normal network perf tests perform between those hosts?
[11:38] <nikbor> IcePic: 20gbit
[11:38] <IcePic> oh sorry, meant to write it as: "did you run netperf on the exact same host setup"
[11:38] <nikbor> yes
[11:39] <nikbor> i even tried switching from infiniband to ethernet - problem persisted
[11:39] <nikbor> this excludes the network, tried changing kernel versions - same thing
[11:41] <nikbor> let's see what blktrace would show
[11:41] * krypto (~krypto@G68-90-105-253.sbcis.sbc.com) Quit (Read error: Connection reset by peer)
[11:41] <mistur> nikbor: try to downgrade to kernel 4.2
[11:41] * krypto (~krypto@106.51.29.212) has joined #ceph
[11:42] <Gugge-47527> nikbor: what queue depth do you test with?
[11:42] <mistur> did you try "ceph tell osd.* bench" ? to see if there is specific osd with poor performance ?
[11:42] <nikbor> fio --name=random-writers --ioengine=libaio --rw=randwrite --direct=1 --size=100m --numjobs=1 --time_based=1 --runtime=100 --iodepth=256 --bs=4k
[11:43] <IcePic> http://tracker.ceph.com/projects/ceph/wiki/Benchmark_Ceph_Cluster_Performance
[11:43] <IcePic> that page had a few good hints on looking at the various levels to see if you can find which part is the slow one.
[11:44] <Gugge-47527> nikbor: i would start with the rbd ioengine, and no filesystem on the rbd
[11:46] <nikbor> mistur: http://pastie.org/private/lkua37nbmia60w0bynm7q according to this all of the osds are pretty close to one another
[11:47] * kefu (~kefu@114.92.96.253) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[11:48] <mistur> nikbor: you have journal on SSD ?
[11:48] * b0e (~aledermue@213.95.25.82) Quit (Ping timeout: 480 seconds)
[11:49] <nikbor> mistur: yep
[11:50] * toMeloos (~toMeloos@53568B3D.cm-6-7c.dynamic.ziggo.nl) has joined #ceph
[11:51] <mistur> the performance are quite good for your osd
[11:53] * b0e (~aledermue@213.95.25.82) has joined #ceph
[11:54] <mistur> nikbor: try other benchmark tools from http://tracker.ceph.com/projects/ceph/wiki/Benchmark_Ceph_Cluster_Performance
[11:55] <mistur> that should help you to identify where is the bottleneck
[11:55] * ira (~ira@c-24-34-255-34.hsd1.ma.comcast.net) has joined #ceph
[11:56] <nikbor> http://pastie.org/private/rfdslnh8yjku9hx4ehmaeq
[11:56] <nikbor> that's trange
[11:56] <nikbor> the average mb per sec keep decreasing
[11:59] * InIMoeK (~InIMoeK@95.170.93.16) has joined #ceph
[12:00] * Pulp (~Pulp@63-221-50-195.dyn.estpak.ee) has joined #ceph
[12:01] <nikbor> rbd bench, sequential is a lot better: http://pastie.org/private/eunnyimypu6tqp22n2vrpq
[12:03] <nikbor> And this is for rand request: http://pastie.org/private/rr5fdua3qw2ptacznmnha
[12:05] * branto (~branto@178-253-133-229.3pp.slovanet.sk) Quit (Ping timeout: 480 seconds)
[12:05] <nikbor> running from a different node i observe similar behavior but the drop stops at 5k iops, and on this server it stops at 200
[12:05] <nikbor> puzzled
[12:06] * InIMoeK (~InIMoeK@95.170.93.16) Quit ()
[12:09] <mistur> http://pastie.org/private/xhtiisf2ep2skkiy1aq8xg
[12:09] <mistur> from my cluster, 70 OSD on 10 Nodes, 10Gb/s networks
[12:10] <mistur> infernalis / ubuntu 14.04 / kernel 3.13.0-76
[12:12] * boredatwork (~overonthe@199.68.193.62) Quit (Read error: Connection reset by peer)
[12:12] * dynamicudpate (~overonthe@199.68.193.62) Quit (Write error: connection closed)
[12:13] <nikbor> so yeah, you are doing linterate
[12:15] <mistur> linterate ?
[12:15] <nikbor> line rate that is
[12:15] * branto (~branto@178-253-133-229.3pp.slovanet.sk) has joined #ceph
[12:15] <nikbor> 10g line rate
[12:15] <mistur> ok
[12:16] <nikbor> and this is on hd, givne the amount of iops you are doing
[12:17] <mistur> I have journal on SSD too (DC S3500)
[12:18] <nikbor> yep
[12:18] <nikbor> hm, i'm really confused by those results to be honest
[12:18] <nikbor> clearly there is something server specific
[12:18] <nikbor> but i cannot find it what, it's not the network, it's not the kernel o_O
[12:19] <mistur> try to downgrade to kernel 4.2
[12:19] <nikbor> i really don't want to do that
[12:19] <nikbor> because in productino there will ben o way of doing that
[12:19] <mistur> I found performance issu on kernel 4.4
[12:19] <mistur> like 50% less than 4.2
[12:20] <mistur> but it was on "ceph tell osd.* bench" where I found this issue
[12:20] <mistur> wiht journal on disk and not ssd
[12:21] <mistur> so yeah maybe is not applyable to your case
[12:22] <nikbor> also if it was kernel related then why don't i see this on other nodes which are running this kernel :)
[12:22] * krypto (~krypto@106.51.29.212) Quit (Read error: Connection reset by peer)
[12:23] <mistur> make sense :)
[12:23] * krypto (~krypto@106.51.29.212) has joined #ceph
[12:23] <mistur> I didn't pay attention to this point
[12:23] * ira (~ira@c-24-34-255-34.hsd1.ma.comcast.net) Quit (Remote host closed the connection)
[12:23] <mistur> so it seem to be server related
[12:23] <nikbor> how do i tell fio to user particular dvice?
[12:25] * ira (~ira@c-24-34-255-34.hsd1.ma.comcast.net) has joined #ceph
[12:26] * nikbor (~n.borisov@admins.1h.com) has left #ceph
[12:26] * nikbor (~n.borisov@admins.1h.com) has joined #ceph
[12:26] <nikbor> ops
[12:26] <nikbor> using : fio --name=random-writers --ioengine=rbd --clientname=admin --rbdname=ludnica2 --pool=rbd --rw=randwrite --direct=1 --size=100m --numjobs=1 --time_based=1 --runtime=100 --iodepth=256 --bs=4k
[12:27] <nikbor> also shows poor performance, meaning this is not block-layer related
[12:28] <mistur> I haven't play a lot with fio yet
[12:28] <mistur> so you might know more than me on that tool
[12:29] <mistur> I must go, bbl
[12:30] * toastyde1th (~toast@pool-71-255-253-39.washdc.fios.verizon.net) has joined #ceph
[12:30] * kuku (~kuku@112.203.6.241) has joined #ceph
[12:33] * F|1nt (~F|1nt@host37-212.lan-isdn.imaginet.fr) has joined #ceph
[12:37] * toastydeath (~toast@pool-71-255-253-39.washdc.fios.verizon.net) Quit (Ping timeout: 480 seconds)
[12:37] * EinstCrazy (~EinstCraz@117.136.8.231) Quit (Read error: Connection reset by peer)
[12:42] * cronburg (~cronburg@209-6-121-249.c3-0.arl-ubr1.sbo-arl.ma.cable.rcn.com) Quit (Ping timeout: 480 seconds)
[12:48] <sep> how can i verify that every single object is removed from a osd ? i did mark one osd out, since it was showing lots of smart errors. and df -h have gone down to about 50GB , but does not seem to reduce further. i would like to know that all objects are off the disk before i down and remove it.
[12:49] * F|1nt (~F|1nt@host37-212.lan-isdn.imaginet.fr) Quit (Quit: Be back later ...)
[12:52] * crismike (~kmajk@nat-hq.ext.getresponse.com) has joined #ceph
[12:52] <crismike> hello, how to add more rgw instances for same zone in new jewel active/acive multisite setup?
[12:54] * bniver (~bniver@pool-173-48-58-27.bstnma.fios.verizon.net) Quit (Remote host closed the connection)
[12:55] * allaok (~allaok@machine107.orange-labs.com) has left #ceph
[12:55] * kuku (~kuku@112.203.6.241) Quit (Read error: Connection reset by peer)
[12:56] * kuku (~kuku@112.203.6.241) has joined #ceph
[13:02] * ade (~abradshaw@ip-178-202-26-185.hsi09.unitymediagroup.de) has joined #ceph
[13:05] * kawa2014 (~kawa@89.184.114.246) Quit (Ping timeout: 480 seconds)
[13:12] * rraja (~rraja@121.244.87.117) has joined #ceph
[13:12] * Arcturus (~Bored@46.166.137.249) has joined #ceph
[13:14] * branto (~branto@178-253-133-229.3pp.slovanet.sk) Quit (Ping timeout: 480 seconds)
[13:15] * vicente (~~vicente@125-227-238-55.HINET-IP.hinet.net) Quit (Quit: Leaving)
[13:15] * kawa2014 (~kawa@212.110.41.244) has joined #ceph
[13:16] * karnan (~karnan@121.244.87.117) Quit (Remote host closed the connection)
[13:18] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) has joined #ceph
[13:18] * xarses_ (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) has joined #ceph
[13:19] * kuku (~kuku@112.203.6.241) Quit (Remote host closed the connection)
[13:25] * xarses (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[13:25] * branto (~branto@178-253-133-229.3pp.slovanet.sk) has joined #ceph
[13:27] * crismike (~kmajk@nat-hq.ext.getresponse.com) Quit (Ping timeout: 480 seconds)
[13:30] * Pulp (~Pulp@63-221-50-195.dyn.estpak.ee) Quit (Read error: Connection reset by peer)
[13:31] * cholcombe (~chris@50-206-35-84.infopact.nl) has joined #ceph
[13:35] * kuku (~kuku@112.203.6.241) has joined #ceph
[13:38] <boolman> will the osd reweight force data migration ?
[13:42] <etienneme> sep: If it's out but cluster is health_OK then it should be ok. You could stop the osd and check if you have some degraded objects
[13:42] * Arcturus (~Bored@26XAAAK43.tor-irc.dnsbl.oftc.net) Quit ()
[13:43] <sep> etienneme, since i ran headfirst into the OOMkiller i have a few days of recovery left before i can expect to see HEALTH_OK
[13:43] * chrish (~chengpeng@180.168.126.243) has joined #ceph
[13:44] <etienneme> Then datas are probably moving. You could check network usage
[13:45] <etienneme> if there is no more network traffic from this osd then it's ok (it works if you have 1 osd per server :p)
[13:45] <sep> only one disk out of 30 on the node that i have set out. there is lots and lots of network traffic.
[13:45] * F|1nt (~F|1nt@host37-212.lan-isdn.imaginet.fr) has joined #ceph
[13:45] <etienneme> OK :) it's weird to have 50 GB of datas
[13:46] <sep> my though exactly
[13:46] <etienneme> You could use one of the object stored on the disk and check if it exists on other disks.
[13:48] * krypto (~krypto@106.51.29.212) Quit (Ping timeout: 480 seconds)
[13:49] * b0e (~aledermue@213.95.25.82) Quit (Ping timeout: 480 seconds)
[13:50] * rdas (~rdas@122.168.253.92) Quit (Quit: Leaving)
[13:53] * wjw-freebsd2 (~wjw@smtp.digiware.nl) Quit (Ping timeout: 480 seconds)
[13:57] * F|1nt (~F|1nt@host37-212.lan-isdn.imaginet.fr) Quit (Ping timeout: 480 seconds)
[13:59] * fdmanana (~fdmanana@2001:8a0:6e0c:6601:454c:daa6:f349:3b58) has joined #ceph
[13:59] * bniver (~bniver@71-9-144-29.static.oxfr.ma.charter.com) has joined #ceph
[14:02] * karnan (~karnan@106.206.159.252) has joined #ceph
[14:02] * b0e (~aledermue@213.95.25.82) has joined #ceph
[14:02] * chengpeng__ (~chengpeng@180.168.126.179) has joined #ceph
[14:04] * chengpeng__ (~chengpeng@180.168.126.179) has left #ceph
[14:05] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) Quit (Quit: neurodrone_)
[14:08] * crismike (~kmajk@nat-hq.ext.getresponse.com) has joined #ceph
[14:08] * rotbeard (~redbeard@185.32.80.238) Quit (Quit: Leaving)
[14:09] * chrish (~chengpeng@180.168.126.243) Quit (Ping timeout: 480 seconds)
[14:10] * analbeard (~shw@5.153.255.226) has joined #ceph
[14:22] * thomnico (~thomnico@2a01:e35:8b41:120:447b:2bf:dd19:182c) has joined #ceph
[14:22] * kuku (~kuku@112.203.6.241) Quit (Read error: Connection reset by peer)
[14:23] * kuku (~kuku@112.203.6.241) has joined #ceph
[14:26] * kawa2014 (~kawa@212.110.41.244) Quit (Ping timeout: 480 seconds)
[14:26] * cyphase (~cyphase@000134f2.user.oftc.net) Quit (Ping timeout: 480 seconds)
[14:27] * cyphase (~cyphase@000134f2.user.oftc.net) has joined #ceph
[14:32] * Racpatel (~Racpatel@2601:87:0:24af::f12b) has joined #ceph
[14:32] * rraja (~rraja@121.244.87.117) Quit (Remote host closed the connection)
[14:34] * valeech (~valeech@pool-108-44-162-111.clppva.fios.verizon.net) Quit (Quit: valeech)
[14:38] * shyu (~Frank@218.241.172.114) has joined #ceph
[14:40] * kawa2014 (~kawa@89.184.114.246) has joined #ceph
[14:40] * vikhyat (~vumrao@121.244.87.116) Quit (Quit: Leaving)
[14:40] * karnan (~karnan@106.206.159.252) Quit (Quit: Leaving)
[14:42] * jcsp (~jspray@82-71-16-249.dsl.in-addr.zen.co.uk) Quit (Ping timeout: 480 seconds)
[14:43] * vbellur (~vijay@71.234.224.255) Quit (Ping timeout: 480 seconds)
[14:45] * bniver (~bniver@71-9-144-29.static.oxfr.ma.charter.com) Quit (Remote host closed the connection)
[14:46] * rraja (~rraja@121.244.87.117) has joined #ceph
[14:47] * vimal (~vikumar@121.244.87.116) Quit (Quit: Leaving)
[14:49] * Kitz (~Kitz@admin163-128.hampshire.edu) Quit (Quit: Kitz)
[14:49] * branto (~branto@178-253-133-229.3pp.slovanet.sk) Quit (Ping timeout: 480 seconds)
[14:50] * Nicho1as (~nicho1as@00022427.user.oftc.net) has joined #ceph
[14:53] * jcsp (~jspray@82-71-16-249.dsl.in-addr.zen.co.uk) has joined #ceph
[14:53] * jgornick (~jgornick@2600:3c00::f03c:91ff:fedf:72b4) has joined #ceph
[14:54] * shaunm (~shaunm@74.83.215.100) has joined #ceph
[14:54] * kefu (~kefu@183.193.112.90) has joined #ceph
[14:54] * thomnico (~thomnico@2a01:e35:8b41:120:447b:2bf:dd19:182c) Quit (Ping timeout: 480 seconds)
[14:55] * dneary (~dneary@pool-96-233-46-27.bstnma.fios.verizon.net) has joined #ceph
[14:56] * derjohn_mob (~aj@fw.gkh-setu.de) Quit (Ping timeout: 480 seconds)
[14:56] * kuku (~kuku@112.203.6.241) Quit (Remote host closed the connection)
[14:57] * thomnico (~thomnico@2a01:e35:8b41:120:447b:2bf:dd19:182c) has joined #ceph
[15:00] * cholcombe (~chris@50-206-35-84.infopact.nl) Quit (Ping timeout: 480 seconds)
[15:01] * branto (~branto@178-253-133-229.3pp.slovanet.sk) has joined #ceph
[15:03] * vikhyat (~vumrao@123.252.252.183) has joined #ceph
[15:08] * bniver (~bniver@pool-173-48-58-27.bstnma.fios.verizon.net) has joined #ceph
[15:10] * rakeshgm (~rakesh@121.244.87.117) Quit (Quit: Leaving)
[15:10] * rakeshgm (~rakesh@121.244.87.117) has joined #ceph
[15:13] * dlan (~dennis@116.228.88.131) has joined #ceph
[15:13] * Miouge (~Miouge@109.128.94.173) Quit (Quit: Miouge)
[15:13] * Miouge (~Miouge@109.128.94.173) has joined #ceph
[15:16] * evelu (~erwan@46.231.131.178) Quit (Ping timeout: 480 seconds)
[15:18] * Hemanth (~hkumar_@121.244.87.117) Quit (Ping timeout: 480 seconds)
[15:18] * squizzi (~squizzi@107.13.31.195) Quit (Quit: bye)
[15:18] * squizzi (~squizzi@107.13.31.195) has joined #ceph
[15:20] * dyasny (~dyasny@cable-192.222.152.136.electronicbox.net) has joined #ceph
[15:27] * evelu (~erwan@46.231.131.178) has joined #ceph
[15:27] * rotbeard (~redbeard@185.32.80.238) has joined #ceph
[15:31] * rwheeler (~rwheeler@nat-pool-bos-t.redhat.com) has joined #ceph
[15:36] * cronburg (~cronburg@nat-pool-bos-t.redhat.com) has joined #ceph
[15:37] * rakeshgm (~rakesh@121.244.87.117) Quit (Quit: Leaving)
[15:40] * ircolle (~Adium@166.175.62.92) has joined #ceph
[15:45] * valeech (~valeech@74-93-221-70-WashingtonDC.hfc.comcastbusiness.net) has joined #ceph
[15:47] * salwasser (~Adium@72.246.3.14) has joined #ceph
[15:50] * dnunez (~dnunez@c-73-38-0-185.hsd1.ma.comcast.net) has joined #ceph
[15:50] * debian112 (~bcolbert@c-73-184-103-26.hsd1.ga.comcast.net) has joined #ceph
[15:51] * yanzheng (~zhyan@118.116.114.34) Quit (Quit: This computer has gone to sleep)
[15:51] * mattbenjamin (~mbenjamin@12.118.3.106) has joined #ceph
[15:52] * vbellur (~vijay@nat-pool-bos-u.redhat.com) has joined #ceph
[15:55] * brad_mssw (~brad@66.129.88.50) has joined #ceph
[15:58] * post-factum (~post-fact@vulcan.natalenko.name) Quit (Killed (NickServ (Too many failed password attempts.)))
[15:58] * vimal (~vikumar@114.143.162.49) has joined #ceph
[15:58] * post-factum (~post-fact@vulcan.natalenko.name) has joined #ceph
[16:03] * vanham (~vanham@mail2.mav.com.br) has joined #ceph
[16:07] <vanham> Guys, so, I have two hosts here with 5 OSDs each. I want my pools to have 3 replicas. So, 3 drives, but always on at least two servers. Is it possible?
[16:07] * xarses_ (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[16:08] * DougalJacobs (~skney@213.61.149.100) has joined #ceph
[16:11] * johnavp1989 (~jpetrini@8.39.115.8) has joined #ceph
[16:11] <- *johnavp1989* To prove that you are human, please enter the result of 8+3
[16:13] * analbeard (~shw@5.153.255.226) Quit (Quit: Leaving.)
[16:15] * gregmark (~Adium@68.87.42.115) has joined #ceph
[16:15] <janos> not recommended but should be
[16:15] <janos> your failure domain would be at the osd level instead of host
[16:15] <janos> making the entire thing much less safe
[16:15] <janos> i forget how you'd ensure not all 3 end up on one host
[16:16] <janos> but again, that scenario in general is not something i would ever recommend
[16:18] * vanham (~vanham@mail2.mav.com.br) Quit (Ping timeout: 480 seconds)
[16:18] * vanham (~vanham@187.20.98.214) has joined #ceph
[16:19] * EthanL (~lamberet@cce02cs4035-fa12-z.ams.hpecore.net) has joined #ceph
[16:19] <etienneme> If you can, get a third host.
[16:19] * analbeard (~shw@support.memset.com) has joined #ceph
[16:19] <etienneme> If one of the host crash, you will be unable to write on some osd
[16:19] * valeech_ (~valeech@74-93-221-70-WashingtonDC.hfc.comcastbusiness.net) has joined #ceph
[16:20] * valeech (~valeech@74-93-221-70-WashingtonDC.hfc.comcastbusiness.net) Quit (Read error: Connection reset by peer)
[16:20] * valeech_ is now known as valeech
[16:21] * vata (~vata@207.96.182.162) has joined #ceph
[16:22] * MentalRay (~MentalRay@office-mtl1-nat-146-218-70-69.gtcomm.net) has joined #ceph
[16:23] * EthanL (~lamberet@cce02cs4035-fa12-z.ams.hpecore.net) Quit (Read error: Connection reset by peer)
[16:23] * MentalRay (~MentalRay@office-mtl1-nat-146-218-70-69.gtcomm.net) Quit ()
[16:25] * kefu (~kefu@183.193.112.90) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[16:25] <jklare> Hi, does anybody know if there are any plans on speeding up the radosgw gc process ? For now this seems to be run in just one thread and in our case this is actually sometimes slower than objects are created.
[16:32] * linuxkidd (~linuxkidd@ip70-189-207-54.lv.lv.cox.net) has joined #ceph
[16:33] * kefu (~kefu@114.92.96.253) has joined #ceph
[16:37] * DougalJacobs (~skney@5AEAAAJ8D.tor-irc.dnsbl.oftc.net) Quit ()
[16:38] * joshd1 (~jdurgin@2602:30a:c089:2b0:5c97:3e75:5f88:68be) has joined #ceph
[16:49] * jdillaman (~jdillaman@pool-108-18-97-95.washdc.fios.verizon.net) has joined #ceph
[16:50] * vikhyat_ (~vumrao@1.39.14.129) has joined #ceph
[16:53] <Anticimex> i'm failing to use "rados -p $poolname import < exportfile". man page or --help doesn't help. what's the actually correct syntax?
[16:54] * vikhyat (~vumrao@123.252.252.183) Quit (Ping timeout: 480 seconds)
[16:54] <Anticimex> rados just print the usage info, gives no error
[16:54] * vimal (~vikumar@114.143.162.49) Quit (Ping timeout: 480 seconds)
[16:55] <scheuk> any radosgw experts/engineers in today?
[16:55] * kefu (~kefu@114.92.96.253) Quit (Max SendQ exceeded)
[16:55] * vimal (~vikumar@114.143.164.10) has joined #ceph
[16:55] * xarses (~xarses@64.124.158.100) has joined #ceph
[16:55] * kefu (~kefu@114.92.96.253) has joined #ceph
[16:56] * derjohn_mob (~aj@46.189.28.62) has joined #ceph
[16:56] * sudocat (~dibarra@104-188-116-197.lightspeed.hstntx.sbcglobal.net) has joined #ceph
[17:00] <ceph-ircslackbot> <vdb> @vanham, you can do at most 2 hosts, distributing 2 copies on a single host (across distinct OSDs) and 1 on the other. But depending on your `min_size` you can have your I/O blocked for a bit when the host containing 2 copies vanishes.
[17:00] <ceph-ircslackbot> <vdb> @vanham, "at least 2" is tricky to do. I'd recommend to rather do 3 hosts directly in that case.
[17:00] <ceph-ircslackbot> <vdb> Like the other recommendations made above.
[17:04] * shyu (~Frank@218.241.172.114) Quit (Quit: Leaving)
[17:04] * sudocat (~dibarra@104-188-116-197.lightspeed.hstntx.sbcglobal.net) Quit (Ping timeout: 480 seconds)
[17:07] * wushudoin (~wushudoin@2601:646:8281:cfd:2ab2:bdff:fe0b:a6ee) has joined #ceph
[17:08] * vikhyat (~vumrao@114.143.177.32) has joined #ceph
[17:09] * Bromine (~Crisco@108.61.122.139) has joined #ceph
[17:09] * thomnico (~thomnico@2a01:e35:8b41:120:447b:2bf:dd19:182c) Quit (Quit: Ex-Chat)
[17:11] * analbeard (~shw@support.memset.com) Quit (Quit: Leaving.)
[17:13] * vikhyat_ (~vumrao@1.39.14.129) Quit (Ping timeout: 480 seconds)
[17:14] * blizzow (~jburns@2601:284:8200:e200:7e7a:91ff:fe14:9b91) has joined #ceph
[17:15] * sudocat (~dibarra@192.185.1.20) has joined #ceph
[17:16] * xarses (~xarses@64.124.158.100) Quit (Remote host closed the connection)
[17:16] * xarses (~xarses@64.124.158.100) has joined #ceph
[17:20] * ade (~abradshaw@ip-178-202-26-185.hsi09.unitymediagroup.de) Quit (Quit: Too sexy for his shirt)
[17:21] * cathode (~cathode@50.232.215.114) has joined #ceph
[17:23] * danieagle (~Daniel@179.110.18.161) has joined #ceph
[17:25] * valeech_ (~valeech@74-93-221-70-WashingtonDC.hfc.comcastbusiness.net) has joined #ceph
[17:27] * valeech (~valeech@74-93-221-70-WashingtonDC.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[17:27] * valeech_ is now known as valeech
[17:35] * newbie45 (~kvirc@host217-114-156-249.pppoe.mark-itt.net) has joined #ceph
[17:37] * vimal (~vikumar@114.143.164.10) Quit (Quit: Leaving)
[17:38] * Miouge (~Miouge@109.128.94.173) Quit (Quit: Miouge)
[17:39] * sudocat (~dibarra@192.185.1.20) Quit (Ping timeout: 480 seconds)
[17:39] * Bromine (~Crisco@5AEAAAJ94.tor-irc.dnsbl.oftc.net) Quit ()
[17:39] * Miouge (~Miouge@109.128.94.173) has joined #ceph
[17:39] * ade (~abradshaw@212.144.230.18) has joined #ceph
[17:48] * efirs (~firs@5.128.174.86) has joined #ceph
[17:49] * TMM (~hp@185.5.121.201) Quit (Quit: Ex-Chat)
[17:50] * vbellur (~vijay@nat-pool-bos-u.redhat.com) Quit (Ping timeout: 480 seconds)
[17:50] * branto (~branto@178-253-133-229.3pp.slovanet.sk) Quit (Ping timeout: 480 seconds)
[17:51] * ntpttr_ (~ntpttr@134.134.139.82) has joined #ceph
[17:52] * b0e (~aledermue@213.95.25.82) Quit (Quit: Leaving.)
[17:54] * karnan (~karnan@106.206.159.252) has joined #ceph
[18:01] * vbellur (~vijay@nat-pool-bos-t.redhat.com) has joined #ceph
[18:01] * rnowling (~rnowling@104-186-210-225.lightspeed.milwwi.sbcglobal.net) has joined #ceph
[18:01] <rnowling> hey folks. how can I list the pgs for a given pool? not the number but the pg ids
[18:02] * vata (~vata@207.96.182.162) Quit (Quit: Leaving.)
[18:03] * Miouge (~Miouge@109.128.94.173) Quit (Quit: Miouge)
[18:04] * branto (~branto@178-253-133-229.3pp.slovanet.sk) has joined #ceph
[18:04] * rnowling (~rnowling@104-186-210-225.lightspeed.milwwi.sbcglobal.net) Quit (Read error: Connection reset by peer)
[18:05] * kawa2014 (~kawa@89.184.114.246) Quit (Quit: Leaving)
[18:07] * Miouge (~Miouge@109.128.94.173) has joined #ceph
[18:08] * kefu (~kefu@114.92.96.253) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[18:08] * vikhyat (~vumrao@114.143.177.32) Quit (Quit: Leaving)
[18:12] * branto (~branto@178-253-133-229.3pp.slovanet.sk) Quit (Ping timeout: 480 seconds)
[18:14] * Miouge (~Miouge@109.128.94.173) Quit (Quit: Miouge)
[18:14] * vasu (~vasu@c-73-231-60-138.hsd1.ca.comcast.net) has joined #ceph
[18:16] * sudocat1 (~dibarra@192.185.1.20) has joined #ceph
[18:18] * valeech (~valeech@74-93-221-70-WashingtonDC.hfc.comcastbusiness.net) Quit (Quit: valeech)
[18:22] * ade (~abradshaw@212.144.230.18) Quit (Quit: Too sexy for his shirt)
[18:24] * joshd1 (~jdurgin@2602:30a:c089:2b0:5c97:3e75:5f88:68be) Quit (Quit: Leaving.)
[18:29] * Miouge (~Miouge@109.128.94.173) has joined #ceph
[18:29] * reed (~reed@216.38.134.18) has joined #ceph
[18:31] * davidzlap (~Adium@rrcs-74-87-213-28.west.biz.rr.com) has joined #ceph
[18:32] * Nicho1as (~nicho1as@00022427.user.oftc.net) Quit (Quit: A man from the Far East; using WeeChat 1.5)
[18:42] * dougf (~dougf@96-38-99-179.dhcp.jcsn.tn.charter.com) Quit (Remote host closed the connection)
[18:42] * Miouge (~Miouge@109.128.94.173) Quit (Read error: Connection reset by peer)
[18:42] * Miouge (~Miouge@109.128.94.173) has joined #ceph
[18:46] * dougf (~dougf@96-38-99-179.dhcp.jcsn.tn.charter.com) has joined #ceph
[18:50] * cholcombe (~chris@50-206-35-84.infopact.nl) has joined #ceph
[18:50] * haplo37 (~haplo37@199.91.185.156) has joined #ceph
[18:56] * Miouge (~Miouge@109.128.94.173) Quit (Quit: Miouge)
[18:57] * linuxkidd (~linuxkidd@ip70-189-207-54.lv.lv.cox.net) Quit (Read error: Connection reset by peer)
[18:57] * linuxkidd (~linuxkidd@ip70-189-207-54.lv.lv.cox.net) has joined #ceph
[18:58] * Hemanth (~hkumar_@103.228.221.141) has joined #ceph
[18:59] <mnaser> I have a cluster that is suffering heavily during recovery io.. IOPs drop down significantly, but the cluster should technically still be able to handle the load
[18:59] <mnaser> it's all SSDs
[19:00] <mnaser> dropping to as low as 100 op/s (but spikes up to the 1000 or so ocasionally), versus the ability to do ~11k usually across the entire cluster.. it's not even recovering at that speed either
[19:02] * mykola (~Mikolaj@91.245.76.240) has joined #ceph
[19:04] * efirs1 (~firs@c-50-185-70-125.hsd1.ca.comcast.net) has joined #ceph
[19:08] * cholcombe (~chris@50-206-35-84.infopact.nl) Quit (Ping timeout: 480 seconds)
[19:12] * rotbeard (~redbeard@185.32.80.238) Quit (Quit: Leaving)
[19:14] * ircolle (~Adium@166.175.62.92) Quit (Quit: Leaving.)
[19:16] * hybrid512 (~walid@195.200.189.206) Quit (Remote host closed the connection)
[19:17] * evelu (~erwan@46.231.131.178) Quit (Ping timeout: 480 seconds)
[19:18] * ntpttr_ (~ntpttr@134.134.139.82) Quit (Ping timeout: 480 seconds)
[19:21] * ntpttr_ (~ntpttr@134.134.139.82) has joined #ceph
[19:23] * bene2 (~bene@nat-pool-bos-t.redhat.com) has joined #ceph
[19:27] <mnaser> the recovery just finished and the cluster is back up doing 22k iops with no problem
[19:27] <mnaser> it seems like there's almost something that "blocks" ops during recovery
[19:30] * Pulp (~Pulp@63-221-50-195.dyn.estpak.ee) has joined #ceph
[19:30] * Miouge (~Miouge@109.128.94.173) has joined #ceph
[19:31] * cmrn (~Jones@45.32.232.26) has joined #ceph
[19:35] * hellertime (~Adium@a23-79-238-10.deploy.static.akamaitechnologies.com) has joined #ceph
[19:36] <SamYaple> mnaser: is this that issue with larger rbds drop in iops?
[19:36] <SamYaple> like >20GB volumes drop in speed?
[19:36] <mnaser> SamYaple, i didnt look at the size of the volumes but there are many volumes on this cluster and most of them are >20G
[19:37] <mnaser> but this happens very specifically over recovery. even if there is *one* placement group that has to be recovered, IO will still be slow, the second they're all active, cluster flies at 22k iops
[19:37] <SamYaple> mnaser: what version of ceph?
[19:38] <mnaser> hammer latest (from centos sig)
[19:38] <ceph-ircslackbot> <vdb> @mnaser: What's your size/min_size?
[19:39] <SamYaple> oh look at that a slackbot
[19:39] <SamYaple> if youre recovering and your active drops to below your min_size it will slow like that
[19:39] <SamYaple> @vdb is there a public slack channel for ceph?
[19:39] <mnaser> well...
[19:40] <mnaser> this is embarassing but min_size=2 and size=2 => not a good time
[19:40] <SamYaple> mnaser: that sounds like the issue
[19:40] <mnaser> and i've been searching all this time
[19:40] <ceph-ircslackbot> <vdb> @SamYaple: There is ceph-storage.slack.com.
[19:40] <SamYaple> thanks @vdb
[19:41] <mnaser> only open for the cool kids apparently though SamYaple :-P
[19:41] <mnaser> oh or we can get invited apparently. according to this at least => http://permalink.gmane.org/gmane.comp.file-systems.ceph.user/30539
[19:42] <SamYaple> mnaser: I at one time had a cisco address
[19:42] <SamYaple> its probably closed now.... right?
[19:42] <mnaser> worth a shot
[19:43] <ceph-ircslackbot> <vdb> Replying (don't reply-all) to that email should suffice.
[19:43] <SamYaple> yep. will add to the todo, thanks @vdb
[19:43] <SamYaple> so mnaser you can lower your min_size, but I would raise your size if at all possible
[19:44] <ceph-ircslackbot> <vdb> I am on Slack all the time across multiple devices so this is a super-convenient option for me. :slightly_smiling_face:
[19:44] <SamYaple> yea my company uses slack, i integrate with it where I can too
[19:45] <mnaser> ^^
[19:45] <mnaser> likewise
[19:45] * sudocat1 (~dibarra@192.185.1.20) Quit (Ping timeout: 480 seconds)
[19:45] * crismike (~kmajk@nat-hq.ext.getresponse.com) Quit (Ping timeout: 480 seconds)
[19:45] * Jeffrey4l_ (~Jeffrey@121.16.111.97) Quit (Ping timeout: 480 seconds)
[19:46] * derjohn_mob (~aj@46.189.28.62) Quit (Ping timeout: 480 seconds)
[19:46] * davidzlap (~Adium@rrcs-74-87-213-28.west.biz.rr.com) Quit (Quit: Leaving.)
[19:48] * Miouge (~Miouge@109.128.94.173) Quit (Quit: Miouge)
[19:51] * efirs (~firs@5.128.174.86) Quit (Read error: No route to host)
[19:51] * efirs (~firs@5.128.174.86) has joined #ceph
[19:53] * sebastian-w_ (~quassel@212.218.8.138) has joined #ceph
[19:54] * karnan (~karnan@106.206.159.252) Quit (Ping timeout: 480 seconds)
[19:55] * rwheeler (~rwheeler@nat-pool-bos-t.redhat.com) Quit (Quit: Leaving)
[19:56] * Pulp (~Pulp@63-221-50-195.dyn.estpak.ee) Quit (Read error: Connection reset by peer)
[19:57] * sebastian-w (~quassel@212.218.8.139) Quit (Ping timeout: 480 seconds)
[19:59] * sebastian-w (~quassel@212.218.8.139) has joined #ceph
[19:59] * ircolle (~Adium@166.175.62.92) has joined #ceph
[20:01] * valeech (~valeech@50-205-143-162-static.hfc.comcastbusiness.net) has joined #ceph
[20:01] * cmrn (~Jones@5AEAAAKDC.tor-irc.dnsbl.oftc.net) Quit ()
[20:02] * sebastian-w_ (~quassel@212.218.8.138) Quit (Ping timeout: 480 seconds)
[20:03] * efirs (~firs@5.128.174.86) Quit (Ping timeout: 480 seconds)
[20:07] * dneary (~dneary@pool-96-233-46-27.bstnma.fios.verizon.net) Quit (Ping timeout: 480 seconds)
[20:11] * codice (~toodles@75-128-34-237.static.mtpk.ca.charter.com) has joined #ceph
[20:16] * wjw-freebsd (~wjw@smtp.digiware.nl) has joined #ceph
[20:16] * sudocat1 (~dibarra@192.185.1.20) has joined #ceph
[20:17] * hellertime (~Adium@a23-79-238-10.deploy.static.akamaitechnologies.com) has left #ceph
[20:21] * valeech (~valeech@50-205-143-162-static.hfc.comcastbusiness.net) Quit (Quit: valeech)
[20:24] * keeperandy (~textual@50.245.231.209) has joined #ceph
[20:30] * Hemanth (~hkumar_@103.228.221.141) Quit (Ping timeout: 480 seconds)
[20:32] * vata (~vata@cable-173.246.3-246.ebox.ca) has joined #ceph
[20:39] * rraja (~rraja@121.244.87.117) Quit (Quit: Leaving)
[20:40] * crismike (~kmajk@host-185-78-133-232.jmdi.pl) has joined #ceph
[20:42] * Peaced (~rapedex@104.156.228.81) has joined #ceph
[20:42] * fdmanana (~fdmanana@2001:8a0:6e0c:6601:454c:daa6:f349:3b58) Quit (Ping timeout: 480 seconds)
[20:53] * ntpttr_ (~ntpttr@134.134.139.82) Quit (Remote host closed the connection)
[21:02] * Peaced (~rapedex@104.156.228.81) Quit (Ping timeout: 480 seconds)
[21:03] <ceph-ircslackbot> <scheuk> hello
[21:03] <scheuk> nice I can talk to mysql :)
[21:03] <scheuk> myself
[21:05] * EinstCrazy (~EinstCraz@180.173.205.135) has joined #ceph
[21:10] * EinstCrazy (~EinstCraz@180.173.205.135) Quit (Remote host closed the connection)
[21:12] * david__ (~david@207.107.71.71) Quit (Quit: Leaving)
[21:12] * david_ (~david@207.107.71.71) Quit (Quit: Leaving)
[21:14] * EinstCrazy (~EinstCraz@180.173.205.135) has joined #ceph
[21:14] * valeech (~valeech@166.170.28.118) has joined #ceph
[21:15] * EinstCrazy (~EinstCraz@180.173.205.135) Quit (Remote host closed the connection)
[21:16] * derjohn_mob (~aj@p578b6aa1.dip0.t-ipconnect.de) has joined #ceph
[21:22] * tZ (~totalworm@static-108-32-49-20.pitbpa.fios.verizon.net) has joined #ceph
[21:24] * mykola (~Mikolaj@91.245.76.240) Quit (Quit: away)
[21:33] * cyphase (~cyphase@000134f2.user.oftc.net) Quit (Ping timeout: 480 seconds)
[21:34] * cyphase (~cyphase@000134f2.user.oftc.net) has joined #ceph
[21:34] * davidzlap (~Adium@2605:e000:1313:8003:7d3a:57b7:282:29ce) has joined #ceph
[21:35] * vata (~vata@cable-173.246.3-246.ebox.ca) Quit (Remote host closed the connection)
[21:37] * Hemanth (~hkumar_@103.228.221.141) has joined #ceph
[21:37] * TomasCZ (~TomasCZ@yes.tenlab.net) has joined #ceph
[21:42] * rendar (~I@host150-177-dynamic.10-87-r.retail.telecomitalia.it) Quit (Ping timeout: 480 seconds)
[21:49] * Hemanth (~hkumar_@103.228.221.141) Quit (Remote host closed the connection)
[21:51] * BrianA (~BrianA@fw-rw.shutterfly.com) has joined #ceph
[21:51] * cholcombe (~chris@50-206-35-84.infopact.nl) has joined #ceph
[21:52] * tZ (~totalworm@9YSAAAVW5.tor-irc.dnsbl.oftc.net) Quit ()
[21:52] * cronburg (~cronburg@nat-pool-bos-t.redhat.com) Quit (Ping timeout: 480 seconds)
[22:04] * cronburg (~cronburg@nat-pool-bos-t.redhat.com) has joined #ceph
[22:05] * Miouge (~Miouge@109.128.94.173) has joined #ceph
[22:07] * rendar (~I@host150-177-dynamic.10-87-r.retail.telecomitalia.it) has joined #ceph
[22:14] * valeech (~valeech@166.170.28.118) Quit (Quit: valeech)
[22:14] * cronburg (~cronburg@nat-pool-bos-t.redhat.com) Quit (Ping timeout: 480 seconds)
[22:16] * ntpttr_ (~ntpttr@134.134.139.82) has joined #ceph
[22:17] * keeperandy (~textual@50.245.231.209) Quit (Quit: Textual IRC Client: www.textualapp.com)
[22:19] <devicenull> is it normal to have to do 'ceph tell mon.* compact' on like a weekly basis?
[22:23] * cronburg (~cronburg@nat-pool-bos-t.redhat.com) has joined #ceph
[22:24] * aarontc (~aarontc@2001:470:e893::1:1) Quit (Quit: Bye!)
[22:26] <[arx]> i don't even know what that does.
[22:27] <devicenull> fixed
[22:27] <devicenull> er, fixes mon.xxx store is getting too big! 31054 MB >= 15360 MB:
[22:27] * vbellur (~vijay@nat-pool-bos-t.redhat.com) Quit (Quit: Leaving.)
[22:28] <devicenull> I dont really know how to figure out why the store is getting big, and can't find a lot of recent information on it
[22:29] * cholcombe (~chris@50-206-35-84.infopact.nl) Quit (Ping timeout: 480 seconds)
[22:29] * Miouge (~Miouge@109.128.94.173) Quit (Quit: Miouge)
[22:31] * ntpttr__ (~ntpttr@192.55.54.36) has joined #ceph
[22:31] * ntpttr_ (~ntpttr@134.134.139.82) Quit (Remote host closed the connection)
[22:34] <[arx]> i haven't seen that error message before either
[22:46] * sudocat1 (~dibarra@192.185.1.20) Quit (Ping timeout: 480 seconds)
[22:48] * sudocat (~dibarra@192.185.1.20) has joined #ceph
[22:57] * brad_mssw (~brad@66.129.88.50) Quit (Quit: Leaving)
[22:57] * sudocat1 (~dibarra@192.185.1.20) has joined #ceph
[23:00] * sudocat (~dibarra@192.185.1.20) Quit (Ping timeout: 480 seconds)
[23:01] * valeech (~valeech@166.170.28.118) has joined #ceph
[23:03] * haplo37 (~haplo37@199.91.185.156) Quit (Remote host closed the connection)
[23:03] * ircolle (~Adium@166.175.62.92) Quit (Quit: Leaving.)
[23:06] <ceph-ircslackbot> <mnaser> Do I have to run our monitoring @ root to get access to the admin socket? :(
[23:08] * davidzlap (~Adium@2605:e000:1313:8003:7d3a:57b7:282:29ce) Quit (Read error: No route to host)
[23:09] <ceph-ircslackbot> <vdb> @mnaser: Which version of Ceph are you using?
[23:09] <ceph-ircslackbot> <mnaser> @vdb: Hammer right now
[23:09] <ceph-ircslackbot> <mnaser> From the CentOS 7 SIG repos
[23:10] <ceph-ircslackbot> <vdb> @mnaser: If you are using Jewel+ you can just add your monitoring user to `ceph` group and that should do it.
[23:10] <ceph-ircslackbot> <mnaser> Aaaah so OSDs are no longer running in root then
[23:10] <ceph-ircslackbot> <vdb> Correct.
[23:10] <ceph-ircslackbot> <vdb> In Hammer and older you will need to play with the privileges of monitoring user, yes.
[23:11] * davidzlap (~Adium@cpe-172-91-154-245.socal.res.rr.com) has joined #ceph
[23:11] <ceph-ircslackbot> <mnaser> not ideal but I'll work around it till we get up to Jewel
[23:13] * kuku (~kuku@112.203.6.241) has joined #ceph
[23:13] * joshd (~jdurgin@206.169.83.146) Quit (Ping timeout: 480 seconds)
[23:13] * valeech (~valeech@166.170.28.118) Quit (Read error: No route to host)
[23:15] * valeech (~valeech@70.88.158.138) has joined #ceph
[23:15] * kuku (~kuku@112.203.6.241) Quit (Read error: Connection reset by peer)
[23:15] * kuku (~kuku@112.203.6.241) has joined #ceph
[23:17] * valeech (~valeech@70.88.158.138) Quit (Read error: Connection reset by peer)
[23:17] * crismike (~kmajk@host-185-78-133-232.jmdi.pl) Quit (Ping timeout: 480 seconds)
[23:18] * valeech (~valeech@70.88.158.138) has joined #ceph
[23:21] * kuku (~kuku@112.203.6.241) Quit (Remote host closed the connection)
[23:23] * valeech_ (~valeech@166.170.28.118) has joined #ceph
[23:26] * valeech (~valeech@70.88.158.138) Quit (Ping timeout: 480 seconds)
[23:26] * valeech_ is now known as valeech
[23:26] * cathode (~cathode@50.232.215.114) Quit (Quit: Leaving)
[23:34] * valeech (~valeech@166.170.28.118) Quit (Read error: No route to host)
[23:35] * newbie45 (~kvirc@host217-114-156-249.pppoe.mark-itt.net) Quit (Ping timeout: 480 seconds)
[23:37] * valeech (~valeech@70.88.158.138) has joined #ceph
[23:39] * joshd (~jdurgin@66-194-8-225.static.twtelecom.net) has joined #ceph
[23:39] * mattbenjamin (~mbenjamin@12.118.3.106) Quit (Ping timeout: 480 seconds)
[23:42] * blizzow (~jburns@2601:284:8200:e200:7e7a:91ff:fe14:9b91) Quit (Ping timeout: 480 seconds)
[23:44] * valeech_ (~valeech@70.88.158.138) has joined #ceph
[23:45] * valeech (~valeech@70.88.158.138) Quit (Read error: Connection reset by peer)
[23:45] * valeech_ is now known as valeech
[23:51] * blizzow (~jburns@2601:284:8200:e200:7e7a:91ff:fe14:9b91) has joined #ceph
[23:57] * danieagle (~Daniel@179.110.18.161) Quit (Quit: Obrigado por Tudo! :-) inte+ :-))

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.