#ceph IRC Log

Index

IRC Log for 2016-07-06

Timestamps are in GMT/BST.

[0:03] <[arx]> isn't a hard thing to do via salt stack
[0:06] * fdmanana (~fdmanana@2001:8a0:6e0c:6601:9901:29b4:b6c5:3fa9) has joined #ceph
[0:10] * ItsCriminalAFK (~PappI@06SAAEVQ5.tor-irc.dnsbl.oftc.net) Quit ()
[0:10] <[arx]> or puppet now that i think about it
[0:10] * BrianA (~BrianA@c-24-130-77-245.hsd1.ca.comcast.net) has joined #ceph
[0:11] * jermudgeon (~jhaustin@31.207.56.59) has joined #ceph
[0:12] <micw> i deploy with ansible but i have only my mons in ceph.conf
[0:13] * ngoswami_ (~ngoswami@1.39.15.161) has joined #ceph
[0:15] * mattbenjamin (~mbenjamin@12.118.3.106) Quit (Quit: Leaving.)
[0:19] * micw (~micw@p5785DD66.dip0.t-ipconnect.de) Quit (Quit: Leaving)
[0:20] * ngoswami (~ngoswami@1.39.15.161) Quit (Ping timeout: 480 seconds)
[0:26] * johnavp1989 (~jpetrini@8.39.115.8) Quit (Remote host closed the connection)
[0:26] * BrianA (~BrianA@c-24-130-77-245.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[0:29] * horan (~horan@pool-173-59-24-80.phlapa.fios.verizon.net) has joined #ceph
[0:30] * fdmanana (~fdmanana@2001:8a0:6e0c:6601:9901:29b4:b6c5:3fa9) Quit (Ping timeout: 480 seconds)
[0:32] <horan> Anyone have any idea's on a solution for this? Attempts to initialize the monitors fail now on Raspbian Jessie when attempting to create a keyring: key for client.admin exists but cap mds does not match
[0:36] * MentalRay (~MentalRay@office-mtl1-nat-146-218-70-69.gtcomm.net) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[0:43] <badone> horan: http://tracker.ceph.com/issues/16443 may offer a workaround and http://tracker.ceph.com/issues/16255 some additional info
[0:44] * rdias (~rdias@bl7-92-98.dsl.telepac.pt) Quit (Ping timeout: 480 seconds)
[0:53] * cathode (~cathode@50.232.215.114) Quit (Quit: Leaving)
[0:53] * rdias (~rdias@2001:8a0:749a:d01:196a:56f9:f6d0:9708) has joined #ceph
[0:54] * Skaag1 (~lunix@65.200.54.234) Quit (Quit: Leaving.)
[0:56] <horan> badone: Thank you for the help I will check them out.
[0:56] <badone> horan: np
[1:02] <horan> badone: The first bug tracker you posted, the workaround appears to have corrected the issue. Thanks again.
[1:02] <badone> horan: glad I could help
[1:06] * stiopa (~stiopa@cpc73832-dals21-2-0-cust453.20-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[1:09] * xarses (~xarses@64.124.158.100) Quit (Ping timeout: 480 seconds)
[1:14] * mhack (~mhack@24-151-36-149.dhcp.nwtn.ct.charter.com) Quit (Remote host closed the connection)
[1:15] * wes_dillingham (~wes_dilli@209-6-222-74.c3-0.hdp-ubr1.sbo-hdp.ma.cable.rcn.com) has joined #ceph
[1:21] * debian112 (~bcolbert@c-73-184-103-26.hsd1.ga.comcast.net) Quit (Ping timeout: 480 seconds)
[1:22] * Brochacho (~alberto@c-50-141-135-98.hsd1.il.comcast.net) Quit (Quit: Brochacho)
[1:35] * ngoswami_ (~ngoswami@1.39.15.161) Quit (Quit: Leaving)
[1:38] * xarses (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) has joined #ceph
[1:49] * sudocat (~dibarra@192.185.1.20) Quit (Quit: Leaving.)
[1:53] * horan (~horan@pool-173-59-24-80.phlapa.fios.verizon.net) Quit (Remote host closed the connection)
[1:53] * rendar (~I@host45-183-dynamic.49-79-r.retail.telecomitalia.it) Quit (Quit: std::lower_bound + std::less_equal *works* with a vector without duplicates!)
[1:57] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) has joined #ceph
[1:58] <TheSov> i just had a very disappoint RL test of a 3 node ceph cluster :(
[1:59] <TheSov> running 2 copies, 1 went down all traffic came to a halt
[1:59] * danieagle (~Daniel@177.94.139.55) Quit (Quit: Obrigado por Tudo! :-) inte+ :-))
[2:00] * wushudoin_ (~wushudoin@2601:646:8281:cfd:2ab2:bdff:fe0b:a6ee) Quit (Quit: Leaving)
[2:00] * Jeffrey4l_ (~Jeffrey@110.244.236.63) has joined #ceph
[2:01] <koszik> it must be the monitors, right? :)
[2:08] <TheSov> one out of 3 yes
[2:08] <TheSov> i made the terrible mistake of merging monitors and osds
[2:09] * linuxkidd (~linuxkidd@ip70-189-207-54.lv.lv.cox.net) Quit (Quit: Leaving)
[2:11] * Racpatel (~Racpatel@2601:87:0:24af::4c8f) Quit (Ping timeout: 480 seconds)
[2:15] * wes_dillingham (~wes_dilli@209-6-222-74.c3-0.hdp-ubr1.sbo-hdp.ma.cable.rcn.com) Quit (Quit: wes_dillingham)
[2:16] * wes_dillingham (~wes_dilli@209-6-222-74.c3-0.hdp-ubr1.sbo-hdp.ma.cable.rcn.com) has joined #ceph
[2:22] * reed (~reed@216.38.134.18) Quit (Quit: Ex-Chat)
[2:24] * bjornar_ (~bjornar@ti0099a430-0410.bb.online.no) Quit (Ping timeout: 480 seconds)
[2:25] * vata (~vata@cable-173.246.3-246.ebox.ca) has joined #ceph
[2:33] <TheSov> ok it seems i broke my ceph cluster
[2:33] <TheSov> how does know what to mount where?
[2:38] <TheSov> i dont get this fuckery, i reboot, the system never mounts the osd locations to /var/lib/ceph/osd/...
[2:39] <TheSov> so i mounted them manually and i did a start ceph-osd id=X
[2:39] <TheSov> it works now but im glad i already knew which disks were which, in the future how would i find this out?
[2:44] <[arx]> did ceph-disk list not show the osd numbers?
[2:45] * Nixx_ (~quassel@bulbasaur.sjorsgielen.nl) Quit (Ping timeout: 480 seconds)
[2:49] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) Quit (Quit: neurodrone_)
[2:51] * Nixx (~quassel@bulbasaur.sjorsgielen.nl) has joined #ceph
[2:55] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) has joined #ceph
[2:58] <TheSov> i dont know i could reboot and find out
[2:58] <TheSov> but if the disk didnt mount and osd didnt start, should ceph disk list show me that information?
[2:59] <TheSov> i just rebooted it
[2:59] <TheSov> lets see
[3:00] <[arx]> i honestly haven't tried it
[3:00] <TheSov> i just dont get the deal here my centos systems never do this
[3:00] <TheSov> just ubuntu
[3:03] * vasu (~vasu@c-73-231-60-138.hsd1.ca.comcast.net) Quit (Quit: Leaving)
[3:04] <badone> TheSov: something to do with the udev rules on Ubuntu?
[3:04] * jermudgeon (~jhaustin@31.207.56.59) Quit (Quit: jermudgeon)
[3:05] <TheSov> it does
[3:05] <TheSov> ceph-disk list shows it even when the osd is not operational as long as it can talk to cluster
[3:05] <TheSov> badone, I would agree but sometimes it works
[3:05] <TheSov> its like bad timing or something
[3:07] <TheSov> funny, my etc/udev/rules.d/ is empty
[3:11] <TheSov> does anyone know where ceph for debian runs it mount command?
[3:11] <TheSov> its not in fstab
[3:11] <m0zes> it should be a udev rule.
[3:11] <TheSov> my /etc/udev/rules.d is empty
[3:12] <TheSov> on all the boxes, even the working ones
[3:13] <TheSov> looks like on ubuntu it uses systemd
[3:13] <TheSov> great
[3:13] <TheSov> i never learned it
[3:13] <m0zes> /usr/lib/udev/rules.d/95-ceph-osd.rules ?
[3:13] <TheSov> nope
[3:13] <TheSov> theres no udev in usr/lib
[3:14] <TheSov> wtf
[3:14] * georgem (~Adium@45.72.156.229) has joined #ceph
[3:14] <m0zes> no clue, I just know it is there for centos...
[3:17] * Racpatel (~Racpatel@2601:87:0:24af::4c8f) has joined #ceph
[3:17] <TheSov> well i mean as long i know how to get them i can script it
[3:17] <TheSov> but thats still bizarre
[3:18] <TheSov> can you show me your udev rules so i can put some in?
[3:19] * boredatwork (~overonthe@199.68.193.62) Quit (Read error: Connection reset by peer)
[3:20] * boredatwork (~overonthe@199.68.193.62) has joined #ceph
[3:20] * Racpatel (~Racpatel@2601:87:0:24af::4c8f) Quit ()
[3:27] * yanzheng (~zhyan@125.70.20.240) has joined #ceph
[3:33] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) Quit (Quit: neurodrone_)
[3:35] * huangjun (~kvirc@117.151.50.153) has joined #ceph
[3:40] <[arx]> what release of ceph are you running?
[3:44] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) has joined #ceph
[3:44] * davidzlap (~Adium@2605:e000:1313:8003:11df:e84:5ff5:43a4) Quit (Quit: Leaving.)
[3:49] * davidzlap (~Adium@2605:e000:1313:8003:11df:e84:5ff5:43a4) has joined #ceph
[3:50] * davidzlap (~Adium@2605:e000:1313:8003:11df:e84:5ff5:43a4) Quit ()
[3:54] * kefu (~kefu@183.193.161.118) has joined #ceph
[3:55] * kefu_ (~kefu@114.92.118.31) has joined #ceph
[3:57] * nojha_ (~nojha@2601:646:8a00:8ab1:8755:db62:bde:a798) Quit (Quit: Leaving)
[3:57] * neha (~nojha@2601:646:8a00:8ab1:8755:db62:bde:a798) Quit (Quit: Leaving)
[4:02] * kefu (~kefu@183.193.161.118) Quit (Ping timeout: 480 seconds)
[4:04] * kefu_ (~kefu@114.92.118.31) Quit (Max SendQ exceeded)
[4:04] * EinstCrazy (~EinstCraz@58.247.119.250) has joined #ceph
[4:04] * kefu (~kefu@114.92.118.31) has joined #ceph
[4:05] * shyu (~Frank@218.241.172.114) has joined #ceph
[4:21] * EinstCrazy (~EinstCraz@58.247.119.250) Quit (Remote host closed the connection)
[4:23] * praveen (~praveen@122.172.223.47) Quit (Read error: Connection reset by peer)
[4:25] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) Quit (Quit: neurodrone_)
[4:25] * MentalRay (~MentalRay@LPRRPQ1401W-LP130-02-1242363207.dsl.bell.ca) has joined #ceph
[4:26] * EinstCrazy (~EinstCraz@60-249-152-164.HINET-IP.hinet.net) has joined #ceph
[4:28] <TheSov> [arx], jewel
[4:28] * neurodrone_ (~neurodron@162.243.191.67) has joined #ceph
[4:28] <TheSov> 10.2.2
[4:29] <Chris_ke> produce environment ?
[4:34] * Sun7zu (~Doodlepie@8b.67.b9d8.ip4.static.sl-reverse.com) has joined #ceph
[4:40] * praveen (~praveen@122.172.223.47) has joined #ceph
[4:44] * ira (~ira@c-24-34-255-34.hsd1.ma.comcast.net) Quit (Ping timeout: 480 seconds)
[4:45] * debian112 (~bcolbert@c-73-184-103-26.hsd1.ga.comcast.net) has joined #ceph
[4:49] * wes_dillingham (~wes_dilli@209-6-222-74.c3-0.hdp-ubr1.sbo-hdp.ma.cable.rcn.com) Quit (Quit: wes_dillingham)
[4:49] * georgem (~Adium@45.72.156.229) Quit (Quit: Leaving.)
[4:49] * MentalRay (~MentalRay@LPRRPQ1401W-LP130-02-1242363207.dsl.bell.ca) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[4:53] * neurodrone_ (~neurodron@162.243.191.67) Quit (Quit: neurodrone_)
[4:54] * MentalRay (~MentalRay@LPRRPQ1401W-LP130-02-1242363207.dsl.bell.ca) has joined #ceph
[4:54] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) has joined #ceph
[4:57] * MentalRay (~MentalRay@LPRRPQ1401W-LP130-02-1242363207.dsl.bell.ca) Quit ()
[4:57] * MentalRay (~MentalRay@LPRRPQ1401W-LP130-02-1242363207.dsl.bell.ca) has joined #ceph
[4:58] * MentalRay (~MentalRay@LPRRPQ1401W-LP130-02-1242363207.dsl.bell.ca) Quit ()
[5:00] * MentalRay (~MentalRay@LPRRPQ1401W-LP130-02-1242363207.dsl.bell.ca) has joined #ceph
[5:01] * MentalRay (~MentalRay@LPRRPQ1401W-LP130-02-1242363207.dsl.bell.ca) Quit ()
[5:04] * Sun7zu (~Doodlepie@8b.67.b9d8.ip4.static.sl-reverse.com) Quit ()
[5:08] * Ceph-Log-Bot (~logstash@2a00:f10:121:400:4da:18ff:fe00:14d0) has joined #ceph
[5:08] * Ceph-Log-Bot (~logstash@2a00:f10:121:400:4da:18ff:fe00:14d0) Quit (Read error: Connection reset by peer)
[5:17] * ahmeni (~Chrissi_@46.166.188.232) has joined #ceph
[5:18] * wes_dillingham (~wes_dilli@209-6-222-74.c3-0.hdp-ubr1.sbo-hdp.ma.cable.rcn.com) has joined #ceph
[5:21] * MentalRay (~MentalRay@LPRRPQ1401W-LP130-02-1242363207.dsl.bell.ca) has joined #ceph
[5:25] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) Quit (Quit: neurodrone_)
[5:30] * rony (~rony@125-227-147-112.HINET-IP.hinet.net) Quit (Remote host closed the connection)
[5:32] * praveen (~praveen@122.172.223.47) Quit (Remote host closed the connection)
[5:37] * Chris_ke (~ke@180.168.170.2) has left #ceph
[5:38] * Chris_ke (~ke@180.168.170.2) has joined #ceph
[5:38] * theTrav (~theTrav@CPE-124-188-218-238.sfcz1.cht.bigpond.net.au) Quit (Remote host closed the connection)
[5:38] * Chris_ke (~ke@180.168.170.2) Quit (Quit: Leaving)
[5:39] * Vacuum__ (~Vacuum@88.130.205.28) has joined #ceph
[5:39] * chengpeng (~chengpeng@180.168.197.98) has joined #ceph
[5:44] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) has joined #ceph
[5:46] <[arx]> TheSov: http://ix.io/10Oz
[5:46] * Vacuum_ (~Vacuum@88.130.200.110) Quit (Ping timeout: 480 seconds)
[5:47] * ahmeni (~Chrissi_@46.166.188.232) Quit ()
[5:49] <TheSov> [arx], thanks
[5:56] * [0x4A6F]_ (~ident@p549C7CC6.dip0.t-ipconnect.de) has joined #ceph
[5:58] * [0x4A6F] (~ident@0x4a6f.user.oftc.net) Quit (Ping timeout: 480 seconds)
[5:58] * [0x4A6F]_ is now known as [0x4A6F]
[6:08] * nwf_ (~nwf@172.56.23.33) Quit (Read error: Connection reset by peer)
[6:09] * nwf_ (~nwf@172.56.23.33) has joined #ceph
[6:10] * IvanJobs (~ivanjobs@103.50.11.146) has joined #ceph
[6:15] * EthanL (~lamberet@cce02cs4036-fa12-z.ams.hpecore.net) Quit (Ping timeout: 480 seconds)
[6:17] * vata (~vata@cable-173.246.3-246.ebox.ca) Quit (Quit: Leaving.)
[6:17] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) Quit (Quit: neurodrone_)
[6:25] * EthanL (~lamberet@cce02cs4036-fa12-z.ams.hpecore.net) has joined #ceph
[6:32] * penguinRaider (~KiKo@69.163.33.182) Quit (Ping timeout: 480 seconds)
[6:40] * wes_dillingham (~wes_dilli@209-6-222-74.c3-0.hdp-ubr1.sbo-hdp.ma.cable.rcn.com) Quit (Quit: wes_dillingham)
[6:45] * penguinRaider (~KiKo@69.163.33.182) has joined #ceph
[6:48] * theTrav (~theTrav@CPE-124-188-218-238.sfcz1.cht.bigpond.net.au) has joined #ceph
[6:50] * spgriffinjr (~spgriffin@66.46.246.206) Quit (Read error: Connection reset by peer)
[6:57] * gauravbafna (~gauravbaf@49.38.1.104) has joined #ceph
[7:01] * arbrandes (~arbrandes@ec2-54-172-54-135.compute-1.amazonaws.com) Quit (Ping timeout: 480 seconds)
[7:04] * jermudgeon (~jhaustin@31.207.56.59) has joined #ceph
[7:05] * penguinRaider (~KiKo@69.163.33.182) Quit (Ping timeout: 480 seconds)
[7:10] * MentalRay (~MentalRay@LPRRPQ1401W-LP130-02-1242363207.dsl.bell.ca) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[7:13] * nwf_ (~nwf@172.56.23.33) has left #ceph
[7:13] * rotbeard (~redbeard@185.32.80.238) has joined #ceph
[7:17] * kefu (~kefu@114.92.118.31) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[7:18] * arbrandes (~arbrandes@ec2-54-172-54-135.compute-1.amazonaws.com) has joined #ceph
[7:22] * penguinRaider (~KiKo@69.163.33.182) has joined #ceph
[7:26] * arbrandes (~arbrandes@ec2-54-172-54-135.compute-1.amazonaws.com) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * ffilz (~ffilz@c-76-115-190-27.hsd1.or.comcast.net) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * ffilzwin (~ffilz@c-76-115-190-27.hsd1.or.comcast.net) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * niknakpaddywak (~xander.ni@outbound.lax.demandmedia.com) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * bniver (~bniver@pool-173-48-58-27.bstnma.fios.verizon.net) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * KindOne (~KindOne@0001a7db.user.oftc.net) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * rwheeler (~rwheeler@pool-173-48-195-215.bstnma.fios.verizon.net) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * jmn (~jmn@nat-pool-bos-t.redhat.com) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * dyasny (~dyasny@cable-192.222.152.136.electronicbox.net) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * jcsp (~jspray@82-71-16-249.dsl.in-addr.zen.co.uk) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * pdrakewe_ (~pdrakeweb@cpe-71-74-153-111.neo.res.rr.com) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * borei (~dan@node-1w7jr9qle4x5ix2kjybp8d4fv.ipv6.telus.net) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * Kingrat (~shiny@2605:a000:161a:c0f6:f0de:64b3:64f7:2f96) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * nhm (~nhm@c-50-171-139-246.hsd1.mn.comcast.net) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * lkoranda (~lkoranda@nat-pool-brq-t.redhat.com) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * owasserm (~owasserm@2001:984:d3f7:1:5ec5:d4ff:fee0:f6dc) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * c0dice (~toodles@75-128-34-237.static.mtpk.ca.charter.com) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * ircuser-1 (~Johnny@158.183-62-69.ftth.swbr.surewest.net) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * jluis (~joao@8.184.114.89.rev.vodafone.pt) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * nolan (~nolan@2001:470:1:41:a800:ff:fe3e:ad08) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * jlayton (~jlayton@2606:a000:1125:405b:c5:7ff:fe41:3227) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * espeer (~quassel@phobos.isoho.st) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * overclk (~quassel@139.59.14.231) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * shaon (~shaon@shaon.me) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * essjayhch (sid79416@id-79416.highgate.irccloud.com) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * gtrott (sid78444@id-78444.tooting.irccloud.com) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * benner (~benner@188.166.111.206) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * ElNounch (sid150478@id-150478.ealing.irccloud.com) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * harbie (~notroot@2a01:4f8:211:2344:0:dead:beef:1) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * Larsen (~andreas@2001:67c:578:2::15) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * Bosse (~bosse@erebus.klykken.com) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * evilrob (~evilrob@2600:3c00::f03c:91ff:fedf:1d3d) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * `10` (~10@69.169.91.14) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * _nick (~nick@zarquon.dischord.org) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * Kruge_ (~Anus@198.211.99.93) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * Tene (~tene@173.13.139.236) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * mlovell (~mlovell@69-195-66-94.unifiedlayer.com) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * LiftedKilt (~LiftedKil@dragons.have.mostlyincorrect.info) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * scalability-junk (sid6422@ealing.irccloud.com) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * oliveiradan (~doliveira@137.65.133.10) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * Animazing (~Wut@94.242.217.235) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * stein (~stein@185.56.185.82) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * Georgyo (~georgyo@shamm.as) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * yebyen (~yebyen@martyfunkhouser.csh.rit.edu) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * Karcaw (~evan@71-95-122-38.dhcp.mdfd.or.charter.com) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * bassam (sid154933@id-154933.brockwell.irccloud.com) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * batrick (~batrick@2600:3c00::f03c:91ff:fe96:477b) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * dmanchad (~dmanchad@nat-pool-bos-t.redhat.com) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * lincolnb (~lincoln@c-71-57-68-189.hsd1.il.comcast.net) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * elder_ (sid70526@id-70526.charlton.irccloud.com) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * janos_ (~messy@static-71-176-211-4.rcmdva.fios.verizon.net) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * diegows (~diegows@main.woitasen.com.ar) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * gmoro (~guilherme@193.120.208.221) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * skullone (~skullone@shell.skull-tech.com) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * carter (~carter@li98-136.members.linode.com) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * devicenull (sid4013@ealing.irccloud.com) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * rinek (~o@62.109.134.112) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * SamYaple (~SamYaple@162.209.126.134) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * MK_FG (~MK_FG@00018720.user.oftc.net) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * pasties (~pasties@00021c52.user.oftc.net) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * braderhart (sid124863@braderhart.user.oftc.net) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * jnq (sid150909@0001b7cc.user.oftc.net) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * med (~medberry@71.74.177.250) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * motk (~motk@2600:3c00::f03c:91ff:fe98:51ee) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * mitchty (~quassel@130-245-47-212.rev.cloud.scaleway.com) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * trociny (~mgolub@93.183.239.2) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * icey (~Chris@0001bbad.user.oftc.net) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * dcwangmit01 (~dcwangmit@162-245.23-239.PUBLIC.monkeybrains.net) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * theanalyst (theanalyst@open.source.rocks.my.socks.firrre.com) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * rektide (~rektide@eldergods.com) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * scubacuda (sid109325@0001fbab.user.oftc.net) Quit (resistance.oftc.net beauty.oftc.net)
[7:26] * mitchty_ (~quassel@130-245-47-212.rev.cloud.scaleway.com) has joined #ceph
[7:26] * Kruge (~Anus@198.211.99.93) has joined #ceph
[7:26] * rektide (~rektide@eldergods.com) has joined #ceph
[7:26] * shaon (~shaon@shaon.me) has joined #ceph
[7:26] * Karcaw (~evan@71-95-122-38.dhcp.mdfd.or.charter.com) has joined #ceph
[7:26] * lincolnb (~lincoln@c-71-57-68-189.hsd1.il.comcast.net) has joined #ceph
[7:26] * `10` (~10@69.169.91.14) has joined #ceph
[7:26] * rinek (~o@62.109.134.112) has joined #ceph
[7:26] * trociny (~mgolub@93.183.239.2) has joined #ceph
[7:26] * med (~medberry@71.74.177.250) has joined #ceph
[7:26] * diegows (~diegows@main.woitasen.com.ar) has joined #ceph
[7:26] * codice (~toodles@75-128-34-237.static.mtpk.ca.charter.com) has joined #ceph
[7:26] * nhm (~nhm@c-50-171-139-246.hsd1.mn.comcast.net) has joined #ceph
[7:26] * bniver (~bniver@pool-173-48-58-27.bstnma.fios.verizon.net) has joined #ceph
[7:26] * Kingrat (~shiny@2605:a000:161a:c0f6:f1d1:746f:18b5:8ac7) has joined #ceph
[7:26] * oliveiradan (~doliveira@137.65.133.10) has joined #ceph
[7:26] * rwheeler (~rwheeler@pool-173-48-195-215.bstnma.fios.verizon.net) has joined #ceph
[7:26] * ChanServ sets mode +o nhm
[7:26] * janos_ (~messy@static-71-176-211-4.rcmdva.fios.verizon.net) has joined #ceph
[7:26] * essjayhch (sid79416@2604:8300:100:200b:6667:1:1:3638) has joined #ceph
[7:26] * jluis (~joao@8.184.114.89.rev.vodafone.pt) has joined #ceph
[7:26] * ChanServ sets mode +o jluis
[7:26] * niknakpaddywak (~xander.ni@23.227.28.5) has joined #ceph
[7:26] * benner (~benner@188.166.111.206) has joined #ceph
[7:26] * Animazing (~Wut@94.242.217.235) has joined #ceph
[7:26] * motk (~motk@2600:3c00::f03c:91ff:fe98:51ee) has joined #ceph
[7:26] * overclk (~quassel@2400:6180:100:d0::54:1) has joined #ceph
[7:26] * lkoranda (~lkoranda@213.175.37.10) has joined #ceph
[7:26] * yebyen (~yebyen@129.21.49.95) has joined #ceph
[7:26] * stein (~stein@185.56.185.82) has joined #ceph
[7:26] * Larsen (~andreas@2001:67c:578:2::15) has joined #ceph
[7:26] * Georgyo (~georgyo@2600:3c03::f03c:91ff:feae:505c) has joined #ceph
[7:26] * SamYaple (~SamYaple@162.209.126.134) has joined #ceph
[7:26] * dmanchad (~dmanchad@66.187.233.206) has joined #ceph
[7:26] * espeer (~quassel@41.78.129.253) has joined #ceph
[7:26] * devicenull (sid4013@2604:8300:100:200b:6667:2:0:fad) has joined #ceph
[7:26] * pdrakeweb (~pdrakeweb@cpe-71-74-153-111.neo.res.rr.com) has joined #ceph
[7:26] * borei (~dan@2001:569:74d8:e300:8256:f2ff:fead:93bb) has joined #ceph
[7:26] * gmoro (~guilherme@193.120.208.221) has joined #ceph
[7:26] * ffilzwin (~ffilz@c-76-115-190-27.hsd1.or.comcast.net) has joined #ceph
[7:26] * batrick (~batrick@2600:3c00::f03c:91ff:fe96:477b) has joined #ceph
[7:26] * ircuser-1 (~Johnny@69.62.183.158) has joined #ceph
[7:26] * skullone (~skullone@107.170.239.224) has joined #ceph
[7:26] * owasserm (~owasserm@2001:984:d3f7:1:5ec5:d4ff:fee0:f6dc) has joined #ceph
[7:26] * arbrandes (~arbrandes@54.172.54.135) has joined #ceph
[7:26] * jcsp (~jspray@82.71.16.249) has joined #ceph
[7:26] * KindOne (~KindOne@198.14.205.1) has joined #ceph
[7:26] * jnq (sid150909@highgate.irccloud.com) has joined #ceph
[7:26] * nolan (~nolan@2001:470:1:41:a800:ff:fe3e:ad08) has joined #ceph
[7:26] * ffilz (~ffilz@76.115.190.27) has joined #ceph
[7:26] * evilrob (~evilrob@2600:3c00::f03c:91ff:fedf:1d3d) has joined #ceph
[7:26] * theanalyst (theanalyst@open.source.rocks.my.socks.firrre.com) has joined #ceph
[7:26] * Bosse (~bosse@2a03:b0c0:2:d0::e9:a001) has joined #ceph
[7:26] * dyasny (~dyasny@192.222.152.136) has joined #ceph
[7:26] * harbie (~notroot@2a01:4f8:211:2344:0:dead:beef:1) has joined #ceph
[7:26] * gtrott (sid78444@2604:8300:100:200b:6667:4:1:326c) has joined #ceph
[7:27] * MK_FG (~MK_FG@00018720.user.oftc.net) has joined #ceph
[7:27] * elder_ (sid70526@2604:8300:100:200b:6667:3:1:137e) has joined #ceph
[7:27] * jlayton (~jlayton@107.13.71.30) has joined #ceph
[7:27] * _nick (~nick@zarquon.dischord.org) has joined #ceph
[7:27] * jproulx (~jon@128.30.30.25) has joined #ceph
[7:27] * mlovell (~mlovell@69-195-66-94.unifiedlayer.com) has joined #ceph
[7:27] * ElNounch (sid150478@2604:8300:100:200b:6667:2:2:4bce) has joined #ceph
[7:27] * scalability-junk (sid6422@2604:8300:100:200b:6667:2:0:1916) has joined #ceph
[7:27] * icey (~Chris@pool-74-103-175-25.phlapa.fios.verizon.net) has joined #ceph
[7:27] * bvi (~Bastiaan@185.56.32.1) has joined #ceph
[7:27] * carter (~carter@li98-136.members.linode.com) has joined #ceph
[7:27] * LiftedKilt (~LiftedKil@is.in.the.madhacker.biz) has joined #ceph
[7:28] * jmn (~jmn@nat-pool-bos-t.redhat.com) has joined #ceph
[7:30] * bassam (sid154933@id-154933.brockwell.irccloud.com) has joined #ceph
[7:30] * scubacuda (sid109325@0001fbab.user.oftc.net) has joined #ceph
[7:30] * braderhart (sid124863@braderhart.user.oftc.net) has joined #ceph
[7:31] * dcwangmit01 (~dcwangmit@162-245.23-239.PUBLIC.monkeybrains.net) has joined #ceph
[7:31] * pasties (~pasties@00021c52.user.oftc.net) has joined #ceph
[7:31] * Tene (~tene@173.13.139.236) has joined #ceph
[7:33] * praveen (~praveen@122.172.223.47) has joined #ceph
[7:41] * praveen (~praveen@122.172.223.47) Quit (Ping timeout: 480 seconds)
[7:53] * Be-El (~blinke@nat-router.computational.bio.uni-giessen.de) has joined #ceph
[7:55] * vikhyat (~vumrao@114.143.47.34) has joined #ceph
[8:13] * kawa2014 (~kawa@89.184.114.246) has joined #ceph
[8:16] * jermudgeon_ (~jhaustin@southend.mdu.whitestone.link) has joined #ceph
[8:18] * micw (~micw@ip92346916.dynamic.kabel-deutschland.de) has joined #ceph
[8:18] <micw> hi
[8:18] <micw> is there a command to check if i can switch off an osd without loosing data?
[8:20] * jermudgeon (~jhaustin@31.207.56.59) Quit (Ping timeout: 480 seconds)
[8:20] * jermudgeon_ is now known as jermudgeon
[8:21] <badone> micw: what size are your pools?
[8:21] <micw> 2
[8:22] <badone> and how many osds do you have?
[8:22] <micw> but i'ts in healing process atm
[8:22] <micw> 12
[8:22] <badone> then I'd wait until it's finished healing
[8:22] <micw> will take 2 days ^^
[8:23] <badone> micw: is that an issue, and if so why?
[8:23] * rendar (~I@host241-113-dynamic.51-82-r.retail.telecomitalia.it) has joined #ceph
[8:25] * ade (~abradshaw@dslb-188-102-071-182.188.102.pools.vodafone-ip.de) has joined #ceph
[8:25] * ade (~abradshaw@dslb-188-102-071-182.188.102.pools.vodafone-ip.de) Quit ()
[8:25] * ade (~abradshaw@dslb-188-102-071-182.188.102.pools.vodafone-ip.de) has joined #ceph
[8:31] <micw> badone, i'm cleaning up the cluster a bit. it was my 1st setup with lot of trial and error. yesterday I did a clean setup of one (of three) nodes.
[8:31] <micw> since yesterday it's self-healing
[8:32] <micw> i'd like to re-setup the 2nd node asap
[8:32] <micw> so i tried to switch off some osds there with the effect that some pgs where "down"
[8:33] <micw> if i understand right, this means unavailability of data?
[8:33] <micw> so i started the osds again and set weight to 0
[8:33] <micw> but i have no idea how to check if i can disable it safely
[8:34] <micw> (other than waiting until all is rebalanced)
[8:36] <badone> micw: if weight is zero it will be moving data off
[8:39] * lincolnb (~lincoln@c-71-57-68-189.hsd1.il.comcast.net) Quit (Ping timeout: 480 seconds)
[8:40] <micw> yes, that's the plan. afterwards i can re-setup the node
[8:42] <micw> hm, setting the size to 1 should improve speed of this operation a lot
[8:42] <badone> so you need to wait for it to move the data then...
[8:42] <badone> micw: if you set size to 1 and *accidentally* lose a node, you will definitely lose data since you only have one copy
[8:42] * treenerd_ (~gsulzberg@cpe90-146-148-47.liwest.at) has joined #ceph
[8:43] <micw> i know
[8:43] <micw> and have backup ^^
[8:44] <badone> micw: okay, so in order to check you would need to dump out your pgs and makse sure there is another copy of the pg on one of the other OSDs before you down it
[8:45] <badone> since size is two you need to make sure there will be one live copy somewhere and the best way to do this is while the cluster is stable
[8:45] <badone> if it is recovering the results may change and you may get caught out since data is moving
[8:46] <micw> i see. seems to be almost impossible to do this manually for ~1000 pgs
[8:46] * penguinRaider (~KiKo@69.163.33.182) Quit (Ping timeout: 480 seconds)
[8:47] <badone> micw: how many osds are currently down?
[8:47] <micw> 12/12 up
[8:47] <micw> 4 set to weight 0
[8:47] <micw> other 4 are newly setup and filled atm
[8:48] <badone> and does your crush rule select by host?
[8:48] * vicente (~~vicente@125-227-238-55.HINET-IP.hinet.net) has joined #ceph
[8:48] <micw> seems so. how can i verify this?
[8:49] <badone> look at your crushmap
[8:50] <badone> micw: how active is it, are there lots of writes happening from clients or gateways?
[8:50] <badone> or rbd images?
[8:50] <micw> http://pastebin.com/tWZ4HTuH
[8:50] <micw> no activity atm
[8:51] <badone> that's not your crushmap
[8:52] <badone> micw: just down an osd then and see whether you get incomplete pgs
[8:53] <micw> http://pastebin.com/RtERqf1E
[8:54] <badone> as long as there is a copy somewhere else ceph will make sure there are "size" copies on up OSDs
[8:55] <badone> step chooseleaf firstn 0 type host
[8:55] * penguinRaider (~KiKo@69.163.33.182) has joined #ceph
[8:55] <badone> so, yes, host
[8:55] <micw> i know. in my initial testings i had size=3 and switched off 2 nodes. looked like the data got spread 3 times on the 4 osds on the ramaining node
[8:55] <micw> at least storage volume groth, so i assume that
[8:55] <badone> micw: that shouldn't happen, and we are talking about OSDs, not nodes
[8:56] <badone> it should only put a single copy on a host, not more than one according to your crushmap
[8:57] <badone> the pgs should have shown as degraded
[8:57] <micw> not sure how my initial crushmap was like
[8:58] <micw> i'm new to ceph and playing around with the cluster for a few days
[8:58] * badone nods
[8:58] <badone> micw: what pgs do you currently have that are not active+clean?
[8:58] * lincolnb (~lincoln@c-71-57-68-189.hsd1.il.comcast.net) has joined #ceph
[8:59] <micw> pgmap v574311: 1088 pgs, 3 pools, 5248 GB data, 1340 kobjects
[8:59] <micw> 7158 GB used, 35806 GB / 42964 GB avail
[8:59] <micw> 1316513/2032460 objects misplaced (64.774%)
[8:59] <micw> 592 active+clean
[8:59] <micw> 491 active+remapped+wait_backfill
[8:59] <micw> 4 active+remapped+backfilling
[8:59] <micw> 1 active+clean+scrubbing+deep
[8:59] <badone> micw: they should be fine
[8:59] <badone> micw: see http://docs.ceph.com/docs/master/rados/operations/pg-states/
[9:00] <badone> you don't want to see down
[9:00] <badone> or degraded if possible
[9:00] <micw> i see
[9:00] <micw> degraded disappeared (of course) after setting size to 1
[9:00] <micw> how can you see that there is only one copy per host on my grushmap?
[9:00] <micw> crushmap?
[9:00] <badone> micw: when size was 2 it means it can't find anywhere to put the other copy
[9:01] <badone> micw: because of the line I specified ^
[9:02] <micw> ?
[9:02] <badone> you only have one rule and it says to choose by host
[9:02] <micw> i see
[9:02] <badone> so for each copy it must choose a different host
[9:02] <micw> now i took one oth the nodes with weight 0 down
[9:02] <badone> node or OSD?
[9:02] <micw> oh, osd
[9:02] <micw> ;)
[9:03] <badone> right
[9:03] <micw> i got 82 down+peering for a few seconds
[9:03] <badone> fine if it's transient
[9:03] <micw> yes. disappeared
[9:03] <micw> so i can try the next now?
[9:04] * kefu (~kefu@183.193.161.118) has joined #ceph
[9:04] <badone> during the peering process it will check the logs of all possible candidates to make sure it has up-to-date details of the objects
[9:04] <badone> micw: how many do you want to do at once?
[9:04] <micw> 85 down+peering
[9:04] <micw> 4 osds
[9:04] <micw> that's one ohst
[9:04] <micw> then i can re-setup it and let it run
[9:04] <badone> and how many up hosts are there?
[9:05] <micw> set the size to 2 again
[9:05] * Concubidated (~cube@nat-pool-nrt-t1.redhat.com) has joined #ceph
[9:05] <micw> 3 hosts at all
[9:05] <badone> so there will only be one down?
[9:05] <micw> yes
[9:06] <badone> that should be okay, as long as you don't see pgs down
[9:06] <badone> or incomplete of course, but that shouldn't happen if the cluster is not active
[9:06] <badone> no writes happening
[9:07] <badone> also don't want to see undersized for long either
[9:07] <badone> with two hosts up it should be able to place two copies so you should be okay
[9:07] * shyu_ (~Frank@218.241.172.114) has joined #ceph
[9:08] * shyu (~Frank@218.241.172.114) Quit (Read error: Connection reset by peer)
[9:08] <badone> it would still be better to wait for it to recover after each step though, safer
[9:09] <micw> seems that all is fine after i took the 4 osds down
[9:09] <micw> 1303042/2026821 objects misplaced (64.290%)
[9:09] <micw> 599 active+clean
[9:09] <micw> 485 active+remapped+wait_backfill
[9:09] <micw> 4 active+remapped+backfilling
[9:09] * kefu_ (~kefu@114.92.118.31) has joined #ceph
[9:09] * SH (~oftc-webi@252.146-78-194.adsl-static.isp.belgacom.be) has joined #ceph
[9:09] <badone> micw: looks okay
[9:09] <micw> what means 489 pgs stuck unclean?
[9:09] <badone> where does it say that?
[9:10] <micw> health HEALTH_WARN
[9:10] <micw> 485 pgs backfill_wait
[9:10] <micw> 4 pgs backfilling
[9:10] <micw> 489 pgs stuck unclean
[9:10] <micw> ceph -s
[9:11] <badone> micw: is that number changing?
[9:12] <badone> unclean is backfill_wait + backfilling
[9:12] <badone> is it still making progress recovering?
[9:12] <micw> oh, i have rebootet the host where i took the osds down. not lots are down...
[9:12] <micw> now
[9:12] <micw> 337 down+peering
[9:13] <micw> seems that "ceph osd down" not actually removes it completely
[9:13] <badone> what do you mean by "removes it completely"?
[9:13] <micw> i thought "ceph osd down" is the same as switching it off
[9:14] <badone> you asked if you could down it, you didn't say anything about removing it completely....
[9:14] <micw> ;)
[9:14] <badone> micw: did you verify they were listed as down and out in "ceph osd tree"?
[9:14] <micw> no
[9:14] <badone> they have to time out before they are marked down and out
[9:15] <badone> so is the status changing?
[9:15] <micw> no
[9:15] <micw> the node is rebooted now
[9:16] <micw> but status is still bad
[9:16] * kefu (~kefu@183.193.161.118) Quit (Ping timeout: 480 seconds)
[9:16] <badone> micw: why did you reboot?
[9:16] <micw> to see if i can safely re-setup that node
[9:16] <micw> host
[9:17] <micw> osds are now up again
[9:18] <micw> seems that i really should wait until recovery is done ;-)
[9:18] <badone> if you have three hosts, chooseleaf host and a size of 2 you should be able to down a host
[9:18] <micw> i should but only after recovery
[9:18] <badone> well, what does ceph -s say now?
[9:19] <micw> because I already had a node down yesterday, re-setu-up it and it's still recovering
[9:19] <badone> te output you gave before showed the pgs were remapped so should have been fine
[9:20] <micw> seems that after "ceph osd down osd.0" the osd comes up again
[9:20] <micw> that's why the "down" states disappeared
[9:20] <Hatsjoe> I also have a 3 node setup, and I have noticed with size 3/min size 2, when you take 1 node offline, all PGs will be stuck unclean, and with status "active+remapped", but this solves itself by bringing the 3rd node back up. It behaves this way because it wants to duplicate the data over all 3 nodes, but can only do it on 2.
[9:20] <micw> i did not see this
[9:20] <micw> my mistake ^^
[9:21] <badone> Hatsjoe: right, but micw has size 2
[9:21] <Hatsjoe> Alright, missed that part
[9:21] <badone> micw: could be systemd restarting it
[9:22] <micw> it's debian /sysvinit
[9:22] <badone> micw: shut the osd daemon down
[9:22] <badone> whatever
[9:22] <badone> Hatsjoe: np :)
[9:22] <micw> 81 down+peering
[9:22] <micw> not changing
[9:23] <badone> micw: does "ceph osd tree" show that OSD as down and out?
[9:23] <micw> down but not out
[9:23] <badone> and is it the only one that is down and out?
[9:24] <micw> the only down
[9:24] <badone> right, so it hasn't been marked out yet
[9:24] <badone> has to time out
[9:24] <micw> can i force it manually?
[9:25] * praveen (~praveen@122.172.223.47) has joined #ceph
[9:25] <micw> oh i have daily in a few minutes
[9:25] <dvahlin> "ceph osd out (osdnr)"
[9:25] <micw> i'll be back in a while
[9:26] <micw> it's out now
[9:26] <micw> still 81 down+peering
[9:26] <micw> should i bring it up again?
[9:27] <badone> seems odd, sure bring it back up
[9:28] * dgurtner (~dgurtner@84-73-130-19.dclient.hispeed.ch) has joined #ceph
[9:28] <badone> at some stage you'll prolly want to find out what is stopping it from peering
[9:28] <badone> micw: are you sure *all* of your pools are at least size 2?
[9:28] <badone> none are size 1?
[9:29] <micw> sorry, afk for ~15 minutes
[9:29] <badone> because that would explain it
[9:29] <badone> sure
[9:29] * derjohn_mob (~aj@p578b6aa1.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[9:29] * dgurtner (~dgurtner@84-73-130-19.dclient.hispeed.ch) Quit ()
[9:29] * dgurtner (~dgurtner@84-73-130-19.dclient.hispeed.ch) has joined #ceph
[9:30] * allaok (~allaok@machine107.orange-labs.com) has joined #ceph
[9:30] <SH> Hi All, 2 days ago some of our ceph disks were full (100%). I managed to restart the ceph cluster and everything seemed to be fine. But since then our Linux machines are unable to mount the volumes and I'm unable to format new volumes. I can however create volumes and They are being mapped on the guest vm. I am running an openstack environment and cinder (volume manager) is telling me this: 2016-07-06 06:52:13.614 24745 AUDIT cin
[9:30] <SH> hmm, sorry... seems like I can't post all my text
[9:30] <SH> I'll start again
[9:31] <SH> So in short, I can't format ceph volumes or mount them
[9:31] <SH> Issue started after we had some disks that were full
[9:31] <SH> I managed to restart ceph which is quite healthy now
[9:32] <SH> We can create ceph volumes, but I can't do anything with them
[9:33] * analbeard (~shw@support.memset.com) has joined #ceph
[9:33] * analbeard (~shw@support.memset.com) has left #ceph
[9:33] <SH> Error log on the linux machine on pastebin: http://pastebin.com/t8223FsM
[9:33] * swami1 (~swami@49.38.0.198) has joined #ceph
[9:41] <badone> SH: qemu-kvm?
[9:41] <badone> yes
[9:42] <badone> SH: Do "rbd" commands work from the qemu-kvm host?
[9:42] <badone> rbd info, etc?
[9:43] <SH> badone: yes, they work
[9:44] * jermudgeon (~jhaustin@southend.mdu.whitestone.link) Quit (Quit: jermudgeon)
[9:45] <badone> SH: you could try turning up rados debugging then to see if that gives any ideas
[9:47] <SH> badone: on the qemu-kvm host?
[9:47] <badone> SH: yep, https://paste.fedoraproject.org/388233/46779126/
[9:48] * T1w (~jens@node3.survey-it.dk) has joined #ceph
[9:48] <badone> SH: qemu-kvm will need to be started after you make the change
[9:49] <badone> or the "instance" will need to be started after you make the change
[9:50] * kefu_ (~kefu@114.92.118.31) Quit (Max SendQ exceeded)
[9:50] <SH> badone: ok, the debug logging should be set in ceph.conf?
[9:50] <badone> SH: that's right
[9:50] * Linkmark (~Linkmark@252.146-78-194.adsl-static.isp.belgacom.be) has joined #ceph
[9:50] * DanFoster (~Daniel@office.34sp.com) has joined #ceph
[9:50] <badone> you can remove it after
[9:51] * kefu (~kefu@ec2-54-64-13-168.ap-northeast-1.compute.amazonaws.com) has joined #ceph
[9:51] * sdw (~oftc-webi@252.146-78-194.adsl-static.isp.belgacom.be) has joined #ceph
[9:52] <SH> ok, I'll restart the qemu machine now
[9:52] * huangjun|2 (~kvirc@117.151.50.153) has joined #ceph
[9:55] * derjohn_mob (~aj@fw.gkh-setu.de) has joined #ceph
[9:55] <badone> SH: you just need to start a new VM, not restart the host
[9:55] <SH> I restarted the VM
[9:55] <SH> But I don't get any logging
[9:56] <badone> SH: something's not right then :)
[9:57] <SH> This morning I have put cinder in debug on my control nodes
[9:57] * moon (~moon@217-19-26-201.dsl.cambrium.nl) has joined #ceph
[9:57] <SH> and there I saw that cinder is mapping the volume on my VM
[9:57] <micw> badone, i had played with size=2 and size=1. state is still inconsistent
[9:57] * Concubidated (~cube@nat-pool-nrt-t1.redhat.com) Quit (Quit: Leaving.)
[9:57] <micw> i'm going to wait until recovery is done
[9:58] * huangjun (~kvirc@117.151.50.153) Quit (Ping timeout: 480 seconds)
[9:58] <SH> badone: have logging now
[9:58] <badone> eureka!
[9:58] <badone> micw: sure
[10:00] <badone> SH: so do something that causes the error on the Vms
[10:00] <badone> then check the rbd debug log
[10:00] <SH> http://pastebin.com/B3nNsiYU
[10:01] <SH> I caused the error, but there was no logging at that moment
[10:01] <SH> In the pastebin the log that was created on boot
[10:02] <SH> the mkfs of the volume is hanging on writing blocks
[10:03] * praveen (~praveen@122.172.223.47) Quit (Remote host closed the connection)
[10:05] <SH> badone: every minute or so I get following log too: http://pastebin.com/KWUNLrqR
[10:06] <badone> SH: Nothing obvious there
[10:07] <SH> Weird thing is that the windows vm's that have volumes mounted seem to work
[10:07] <SH> only Linux is impacted
[10:07] <badone> SH: I'd suggest you crank up debug logging for qemu-kvm and see if you can find anything there
[10:08] <SH> ok, I'll post again if I find something
[10:09] <SH> thx for you help :)
[10:09] * rraja (~rraja@121.244.87.117) has joined #ceph
[10:09] <badone> SH: np. I'd also recommend going over the logs on the ceph cluster and see if there are any errors there
[10:10] * rraja (~rraja@121.244.87.117) Quit ()
[10:10] * moon (~moon@217-19-26-201.dsl.cambrium.nl) Quit (Ping timeout: 480 seconds)
[10:17] * SquallSeeD31 (~Tralin|Sl@tsn109-201-154-178.dyn.nltelcom.net) has joined #ceph
[10:18] * linjan (~linjan@176.195.184.236) has joined #ceph
[10:19] * i_m (~ivan.miro@deibp9eh1--blueice4n6.emea.ibm.com) has joined #ceph
[10:20] * Concubidated (~cube@170.106.49.163.rev.iijmobile.jp) has joined #ceph
[10:25] * kellyer (~Thunderbi@dub-bdtn-office-r1.net.digiweb.ie) has joined #ceph
[10:26] * rotbeard (~redbeard@185.32.80.238) Quit (Quit: Leaving)
[10:31] * dan__ (~Daniel@office.34sp.com) has joined #ceph
[10:31] * DanFoster (~Daniel@office.34sp.com) Quit (Read error: Connection reset by peer)
[10:35] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[10:36] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[10:36] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[10:37] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[10:37] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[10:38] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[10:38] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[10:39] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[10:39] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[10:39] * wujson is now known as zokko
[10:40] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[10:40] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[10:41] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[10:41] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[10:42] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[10:42] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[10:43] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[10:43] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[10:44] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[10:44] * TMM (~hp@185.5.121.201) has joined #ceph
[10:44] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[10:45] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[10:45] * praveen (~praveen@121.244.155.9) has joined #ceph
[10:45] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[10:46] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[10:46] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[10:47] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[10:47] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[10:47] * SquallSeeD31 (~Tralin|Sl@06SAAEXAM.tor-irc.dnsbl.oftc.net) Quit ()
[10:48] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[10:48] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[10:49] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[10:49] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[10:50] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[10:50] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[10:50] * nass5 (~fred@l-p-dn-in-12a.lionnois.site.univ-lorraine.fr) Quit (Ping timeout: 480 seconds)
[10:51] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[10:51] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[10:52] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[10:52] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[10:53] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[10:53] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[10:53] * DV (~veillard@2001:41d0:a:f29f::1) Quit (Ping timeout: 480 seconds)
[10:53] * rotbeard (~redbeard@185.32.80.238) has joined #ceph
[10:54] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[10:54] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[10:55] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[10:55] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[10:56] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[10:56] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[10:57] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[10:57] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[10:58] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[10:58] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[10:58] * Concubidated1 (~cube@122.103.163.63.ap.gmobb-fix.jp) has joined #ceph
[10:59] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[10:59] * Concubidated2 (~cube@170.106.49.163.rev.iijmobile.jp) has joined #ceph
[10:59] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[11:00] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[11:00] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[11:00] * Concubidated (~cube@170.106.49.163.rev.iijmobile.jp) Quit (Read error: No route to host)
[11:01] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[11:01] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[11:02] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[11:02] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[11:02] * DV (~veillard@2001:41d0:a:f29f::1) has joined #ceph
[11:03] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[11:03] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[11:04] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[11:04] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[11:05] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[11:05] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[11:06] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[11:06] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[11:06] * chengpeng (~chengpeng@180.168.197.98) has left #ceph
[11:06] * chengpeng (~chengpeng@180.168.197.98) has joined #ceph
[11:06] * Concubidated1 (~cube@122.103.163.63.ap.gmobb-fix.jp) Quit (Ping timeout: 480 seconds)
[11:07] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[11:07] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[11:08] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[11:08] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[11:09] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[11:09] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[11:10] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[11:10] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) has joined #ceph
[11:10] * haomaiwang (~haomaiwan@210-129-17-7.jp-east.compute.idcfcloud.com) Quit (Remote host closed the connection)
[11:12] * haomaiwang (~haomaiwan@li707-115.members.linode.com) has joined #ceph
[11:12] * nass5 (~fred@l-p-dn-in-12a.lionnois.site.univ-lorraine.fr) has joined #ceph
[11:13] * haomaiwang (~haomaiwan@li707-115.members.linode.com) Quit (Remote host closed the connection)
[11:13] * haomaiwang (~haomaiwan@li707-115.members.linode.com) has joined #ceph
[11:13] * fdmanana (~fdmanana@2001:8a0:6e0c:6601:9901:29b4:b6c5:3fa9) has joined #ceph
[11:13] * haomaiwang (~haomaiwan@li707-115.members.linode.com) Quit ()
[11:22] * mgolub (~mgolub@93.183.239.2) has joined #ceph
[11:22] * trociny (~mgolub@93.183.239.2) Quit (Read error: Connection reset by peer)
[11:22] * trociny (~mgolub@93.183.239.2) has joined #ceph
[11:23] * mgolub (~mgolub@93.183.239.2) Quit (Remote host closed the connection)
[11:27] * SamYaple (~SamYaple@162.209.126.134) Quit (Ping timeout: 480 seconds)
[11:33] <Raboo> i'm running a ssd partition as a journal device. Do I need to trim that partition?
[11:34] <Raboo> it's actually a nvme partition
[11:36] * swami2 (~swami@49.44.57.244) has joined #ceph
[11:36] * dugravot6 (~dugravot6@l-p-dn-in-4a.lionnois.site.univ-lorraine.fr) Quit (Quit: Leaving.)
[11:37] * lmb (~Lars@ip5b41f0a4.dynamic.kabel-deutschland.de) Quit (Remote host closed the connection)
[11:40] * swami1 (~swami@49.38.0.198) Quit (Ping timeout: 480 seconds)
[11:40] * bvi (~Bastiaan@185.56.32.1) Quit (Remote host closed the connection)
[11:42] * lmb (~Lars@2a02:8109:8100:1d2c:1df6:7c81:4fcb:7bcf) has joined #ceph
[11:45] * ira (~ira@c-24-34-255-34.hsd1.ma.comcast.net) has joined #ceph
[11:55] * sebastian-w (~quassel@212.218.8.138) Quit (Read error: Connection reset by peer)
[11:55] * sebastian-w (~quassel@212.218.8.138) has joined #ceph
[11:59] * mjevans (~mjevans@li984-246.members.linode.com) Quit (Server closed connection)
[11:59] * mjevans (~mjevans@li984-246.members.linode.com) has joined #ceph
[12:06] * rkeene (1011@oc9.org) Quit (Server closed connection)
[12:06] * rkeene (1011@oc9.org) has joined #ceph
[12:08] * starcoder (~legion@tor-exit4-readme.dfri.se) has joined #ceph
[12:09] * derjohn_mob (~aj@fw.gkh-setu.de) Quit (Ping timeout: 480 seconds)
[12:17] * derjohn_mob (~aj@fw.gkh-setu.de) has joined #ceph
[12:22] * SH (~oftc-webi@252.146-78-194.adsl-static.isp.belgacom.be) Quit (Quit: Page closed)
[12:29] * GeoTracer (~Geoffrey@41.77.153.99) Quit (Ping timeout: 480 seconds)
[12:30] * GeoTracer (~Geoffrey@41.77.153.99) has joined #ceph
[12:32] * The1_ (~the_one@5.186.54.143) Quit (Read error: Connection reset by peer)
[12:34] * swami1 (~swami@49.38.0.153) has joined #ceph
[12:35] * linjan (~linjan@176.195.184.236) Quit (Ping timeout: 480 seconds)
[12:35] * T1 (~the_one@5.186.54.143) has joined #ceph
[12:35] * skarn (skarn@0001f985.user.oftc.net) Quit (Server closed connection)
[12:35] * evelu (~erwan@2a01:e34:eecb:7400:4eeb:42ff:fedc:8ac) Quit (Ping timeout: 480 seconds)
[12:35] * skarn_ (skarn@flame.firrre.com) has joined #ceph
[12:35] * skarn_ is now known as skarn
[12:38] * starcoder (~legion@9YSAAAB3R.tor-irc.dnsbl.oftc.net) Quit ()
[12:39] * jidar (~jidar@r2d2.fap.me) Quit (Server closed connection)
[12:39] * swami2 (~swami@49.44.57.244) Quit (Ping timeout: 480 seconds)
[12:42] * Concubidated (~cube@122.103.163.63.ap.gmobb-fix.jp) has joined #ceph
[12:47] * mischief (~mischief@iota.offblast.org) Quit (Server closed connection)
[12:47] * mischief (~mischief@iota.offblast.org) has joined #ceph
[12:48] * Concubidated2 (~cube@170.106.49.163.rev.iijmobile.jp) Quit (Ping timeout: 480 seconds)
[12:48] * SH (~oftc-webi@252.146-78-194.adsl-static.isp.belgacom.be) has joined #ceph
[12:51] * treenerd_ (~gsulzberg@cpe90-146-148-47.liwest.at) Quit (Quit: treenerd_)
[12:54] * jackhill (~jackhill@bog.hcoop.net) Quit (Server closed connection)
[12:54] * Concubidated (~cube@122.103.163.63.ap.gmobb-fix.jp) Quit (Read error: Connection reset by peer)
[12:54] * jackhill (~jackhill@bog.hcoop.net) has joined #ceph
[12:55] * Concubidated (~cube@122.103.163.63.ap.gmobb-fix.jp) has joined #ceph
[12:55] * mdxi (~mdxi@li925-141.members.linode.com) Quit (Server closed connection)
[12:55] * mdxi (~mdxi@li925-141.members.linode.com) has joined #ceph
[12:56] * theTrav (~theTrav@CPE-124-188-218-238.sfcz1.cht.bigpond.net.au) Quit (Remote host closed the connection)
[12:56] * linjan (~linjan@176.195.184.236) has joined #ceph
[13:04] * sdw (~oftc-webi@252.146-78-194.adsl-static.isp.belgacom.be) Quit (Ping timeout: 480 seconds)
[13:05] * EinstCrazy (~EinstCraz@60-249-152-164.HINET-IP.hinet.net) Quit (Remote host closed the connection)
[13:07] * vicente (~~vicente@125-227-238-55.HINET-IP.hinet.net) Quit (Quit: Leaving)
[13:08] <Linkmark> Is it mandatory to have rbd and libceph kernel modules loaded on osd and or mon nodes?
[13:08] * TMM (~hp@185.5.121.201) Quit (Ping timeout: 480 seconds)
[13:09] <badone> Linkmark: no, you shouldn't need them at all
[13:09] <badone> Linkmark: they are for clients only
[13:10] <Linkmark> yeah i tough so, thanks badone !
[13:10] <badone> np
[13:12] * praveen (~praveen@121.244.155.9) Quit (Server closed connection)
[13:13] * praveen (~praveen@121.244.155.9) has joined #ceph
[13:14] * wjw-freebsd (~wjw@smtp.digiware.nl) Quit (Ping timeout: 480 seconds)
[13:19] * DV (~veillard@2001:41d0:a:f29f::1) Quit (Server closed connection)
[13:19] * DV (~veillard@2001:41d0:a:f29f::1) has joined #ceph
[13:21] * skorgu (skorgu@pylon.skorgu.net) Quit (Server closed connection)
[13:21] * skorgu (skorgu@pylon.skorgu.net) has joined #ceph
[13:25] * linjan (~linjan@176.195.184.236) Quit (Ping timeout: 480 seconds)
[13:25] * debian112 (~bcolbert@c-73-184-103-26.hsd1.ga.comcast.net) Quit (Ping timeout: 480 seconds)
[13:28] * fdmanana (~fdmanana@2001:8a0:6e0c:6601:9901:29b4:b6c5:3fa9) Quit (Server closed connection)
[13:29] * fdmanana (~fdmanana@2001:8a0:6e0c:6601:9901:29b4:b6c5:3fa9) has joined #ceph
[13:29] * huangjun (~kvirc@117.152.65.191) has joined #ceph
[13:29] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) has joined #ceph
[13:30] * trociny (~mgolub@93.183.239.2) Quit (Server closed connection)
[13:30] * trociny (~mgolub@93.183.239.2) has joined #ceph
[13:33] <Hatsjoe> Having a weird issue atm, libvirt cannot connect to my rbd pool "failed to connect to the RADOS monitor on: <IPs and ports here>: Connection timed out" while everything is up and running, not being blocked, and I can telnet to the IP/ports from the libvirt node
[13:33] <Hatsjoe> Anyone having an idea?
[13:34] <Hatsjoe> (I also see traffic with tcpdump between the MONs and the libvirt node)
[13:35] * huangjun|2 (~kvirc@117.151.50.153) Quit (Ping timeout: 480 seconds)
[13:35] * rraja (~rraja@121.244.87.117) has joined #ceph
[13:36] * evelu (~erwan@2a01:e34:eecb:7400:4eeb:42ff:fedc:8ac) has joined #ceph
[13:38] * debian112 (~bcolbert@c-73-184-103-26.hsd1.ga.comcast.net) has joined #ceph
[13:54] * bvi (~Bastiaan@185.56.32.1) has joined #ceph
[13:59] * IvanJobs (~ivanjobs@103.50.11.146) Quit ()
[14:01] <Hatsjoe> Found out the issue, due to different ceph version on the client and cluster, it could not connect
[14:03] * rotbeard (~redbeard@185.32.80.238) Quit (Quit: Leaving)
[14:03] <MaZ-> hmm, manually rebuilt latest packages from master, applied all of the RGW memory leak fixes I can see in open pull requests... still massive memory leak / usage / whatevs
[14:05] <zdzichu> root
[14:05] <zdzichu> sorry, wrong window
[14:08] * TMM (~hp@185.5.121.201) has joined #ceph
[14:10] * debian112 (~bcolbert@c-73-184-103-26.hsd1.ga.comcast.net) has left #ceph
[14:10] * debian112 (~bcolbert@c-73-184-103-26.hsd1.ga.comcast.net) has joined #ceph
[14:11] * HappyLoaf (~HappyLoaf@cpc93928-bolt16-2-0-cust133.10-3.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[14:16] * gauravbafna (~gauravbaf@49.38.1.104) Quit (Remote host closed the connection)
[14:21] * rwheeler (~rwheeler@pool-173-48-195-215.bstnma.fios.verizon.net) Quit (Quit: Leaving)
[14:26] * Aethis (~thundercl@213.61.149.100) has joined #ceph
[14:28] * i_m (~ivan.miro@deibp9eh1--blueice4n6.emea.ibm.com) Quit (Ping timeout: 480 seconds)
[14:36] * kutija (~kutija@89.216.27.139) has joined #ceph
[14:37] * gluco (~gluco@84.88.65.65) has joined #ceph
[14:49] * swami1 (~swami@49.38.0.153) Quit (Quit: Leaving.)
[14:52] * georgem (~Adium@206.108.127.16) has joined #ceph
[14:55] * mhack (~mhack@nat-pool-bos-t.redhat.com) has joined #ceph
[14:58] * mattbenjamin (~mbenjamin@76-206-42-50.lightspeed.livnmi.sbcglobal.net) has joined #ceph
[14:59] * Aethis (~thundercl@4MJAAHHEQ.tor-irc.dnsbl.oftc.net) Quit (Ping timeout: 480 seconds)
[15:03] * EthanL (~lamberet@cce02cs4036-fa12-z.ams.hpecore.net) Quit (Ping timeout: 480 seconds)
[15:03] * billwebb (~billwebb@50-203-47-138-static.hfc.comcastbusiness.net) has joined #ceph
[15:06] * kefu_ (~kefu@114.92.118.31) has joined #ceph
[15:09] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) Quit (Quit: neurodrone_)
[15:09] * kefu (~kefu@ec2-54-64-13-168.ap-northeast-1.compute.amazonaws.com) Quit (Ping timeout: 480 seconds)
[15:11] * spgriffinjr (~spgriffin@66.46.246.206) has joined #ceph
[15:11] * EthanL (~lamberet@cce02cs4036-fa12-z.ams.hpecore.net) has joined #ceph
[15:15] * johnavp1989 (~jpetrini@8.39.115.8) has joined #ceph
[15:15] <- *johnavp1989* To prove that you are human, please enter the result of 8+3
[15:16] * kefu_ (~kefu@114.92.118.31) Quit (Max SendQ exceeded)
[15:16] * kefu (~kefu@114.92.118.31) has joined #ceph
[15:20] * SH (~oftc-webi@252.146-78-194.adsl-static.isp.belgacom.be) Quit (Quit: Page closed)
[15:27] * brad_mssw (~brad@66.129.88.50) has joined #ceph
[15:29] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[15:30] * Racpatel (~Racpatel@2601:87:0:24af::4c8f) has joined #ceph
[15:31] * rwheeler (~rwheeler@nat-pool-bos-t.redhat.com) has joined #ceph
[15:35] * vbel (~oftc-webi@ukc1-proxy-mwg08-o.oracle.com) has joined #ceph
[15:35] * scg (~zscg@146-115-134-246.c3-0.nwt-ubr1.sbo-nwt.ma.cable.rcn.com) has joined #ceph
[15:36] * kellyer1 (~Thunderbi@dub-bdtn-office-r1.net.digiweb.ie) has joined #ceph
[15:36] <vbel> Hi everyone, I am getting problems with slow monitors (3). They fail to join, logs contain a lot of mon.10.20.30.40@0(leader).paxos(paxos recovering c 0..0) collect timeout, calling fresh election. What parameters can I tune?
[15:37] * kellyer (~Thunderbi@dub-bdtn-office-r1.net.digiweb.ie) Quit (Read error: Connection reset by peer)
[15:37] * kellyer1 is now known as kellyer
[15:37] * Heebie (~thebert@dub-bdtn-office-r1.net.digiweb.ie) Quit (Read error: Connection reset by peer)
[15:37] <vbel> i have ntp running, ceph mons recognize each other (no firewalls)
[15:38] * Heebie (~thebert@dub-bdtn-office-r1.net.digiweb.ie) has joined #ceph
[15:39] * shyu_ (~Frank@218.241.172.114) Quit (Ping timeout: 480 seconds)
[15:40] * wes_dillingham (~wes_dilli@140.247.242.44) has joined #ceph
[15:40] * mattbenjamin (~mbenjamin@76-206-42-50.lightspeed.livnmi.sbcglobal.net) Quit (Ping timeout: 480 seconds)
[15:41] * Silentspy (~Rosenblut@185.108.128.8) has joined #ceph
[15:42] * yanzheng (~zhyan@125.70.20.240) Quit (Quit: This computer has gone to sleep)
[15:46] * cr0wrx (~oftc-webi@c-50-180-203-98.hsd1.ga.comcast.net) Quit (Ping timeout: 480 seconds)
[15:51] * kefu (~kefu@114.92.118.31) Quit (Max SendQ exceeded)
[15:52] * kefu (~kefu@114.92.118.31) has joined #ceph
[15:54] * karnan (~karnan@103.227.98.153) has joined #ceph
[15:54] * karnan (~karnan@103.227.98.153) Quit ()
[15:55] * kefu (~kefu@114.92.118.31) Quit (Max SendQ exceeded)
[15:57] * kefu (~kefu@114.92.118.31) has joined #ceph
[16:01] * ntpttr_ (~ntpttr@fmdmzpr03-ext.fm.intel.com) has joined #ceph
[16:02] * xarses (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[16:03] * SamYaple (~SamYaple@162.209.126.134) has joined #ceph
[16:07] * MentalRay (~MentalRay@office-mtl1-nat-146-218-70-69.gtcomm.net) has joined #ceph
[16:08] * SamYaple (~SamYaple@162.209.126.134) Quit ()
[16:08] * SamYaple (~SamYaple@162.209.126.134) has joined #ceph
[16:11] * Silentspy (~Rosenblut@185.108.128.8) Quit ()
[16:12] * Ethan_L (~lamberet@cce02cs4035-fa12-z.ams.hpecore.net) has joined #ceph
[16:12] * mattbenjamin (~mbenjamin@12.118.3.106) has joined #ceph
[16:12] * bjornar_ (~bjornar@ti0099a430-0410.bb.online.no) has joined #ceph
[16:13] * vata (~vata@207.96.182.162) has joined #ceph
[16:13] * EthanL (~lamberet@cce02cs4036-fa12-z.ams.hpecore.net) Quit (Read error: Connection reset by peer)
[16:17] * Concubidated (~cube@122.103.163.63.ap.gmobb-fix.jp) Quit (Read error: Connection reset by peer)
[16:17] * Concubidated (~cube@122.103.163.63.ap.gmobb-fix.jp) has joined #ceph
[16:17] * T1w (~jens@node3.survey-it.dk) Quit (Ping timeout: 480 seconds)
[16:18] * Concubidated1 (~cube@139.26.13.160.rev.iijmobile.jp) has joined #ceph
[16:25] * xarses (~xarses@64.124.158.100) has joined #ceph
[16:25] * Concubidated (~cube@122.103.163.63.ap.gmobb-fix.jp) Quit (Ping timeout: 480 seconds)
[16:27] * cronburg (~cronburg@nat-pool-bos-t.redhat.com) Quit (Ping timeout: 480 seconds)
[16:28] * gauravbafna (~gauravbaf@122.178.243.68) has joined #ceph
[16:32] * rotbeard (~redbeard@2a02:908:df13:bb00:a863:41df:8783:1972) has joined #ceph
[16:32] <vbel> Hi everyone, I am getting problems with slow monitors (3). They fail to make quorum, logs contain a lot of mon.10.20.30.40@0(leader).paxos(paxos recovering c 0..0) collect timeout, calling fresh election. What parameters can I tune?
[16:34] * tallest_red (~djidis__@tor2r.ins.tor.net.eu.org) has joined #ceph
[16:34] * danieagle (~Daniel@201-95-102-169.dsl.telesp.net.br) has joined #ceph
[16:34] <jluis> vbel, are the monitors able to reach each other?
[16:35] <jluis> you should probably enable 'debug ms = 1' and 'debug mon = 10' and check whether the peons are replying to the leader's messages during recovery
[16:36] <jluis> also, check whether there are any clock skews between the monitors
[16:37] * gauravbafna (~gauravbaf@122.178.243.68) Quit (Ping timeout: 480 seconds)
[16:37] * ntpttr_ (~ntpttr@fmdmzpr03-ext.fm.intel.com) Quit (Remote host closed the connection)
[16:39] * joshd1 (~jdurgin@2602:30a:c089:2b0:3082:243e:bc2d:3f72) has joined #ceph
[16:43] * cronburg (~cronburg@nat-pool-bos-t.redhat.com) has joined #ceph
[16:44] * huangjun|2 (~kvirc@117.152.65.191) has joined #ceph
[16:47] * huangjun (~kvirc@117.152.65.191) Quit (Ping timeout: 480 seconds)
[16:53] * Ethan_L (~lamberet@cce02cs4035-fa12-z.ams.hpecore.net) Quit (Ping timeout: 480 seconds)
[16:53] <vbel> jluis: thanks! ntp works well, debug mon = 10 is set, I will add debug ms - 1. I can access monitors via their ports. now: e0 reset_probe_timeout 0x1fe6170 after 2 seconds, probing other monitors ..
[16:54] * wushudoin (~wushudoin@2601:646:8281:cfd:2ab2:bdff:fe0b:a6ee) has joined #ceph
[16:58] * ntpttr_ (~ntpttr@134.134.139.74) has joined #ceph
[17:00] * ceph-ircslackbot (~ceph-ircs@ds9536.dreamservers.com) Quit (Remote host closed the connection)
[17:01] * ceph-ircslackbot (~ceph-ircs@ds9536.dreamservers.com) has joined #ceph
[17:01] * gauravbafna (~gauravbaf@122.178.243.68) has joined #ceph
[17:02] * kawa2014 (~kawa@89.184.114.246) Quit (Ping timeout: 480 seconds)
[17:02] <jluis> vbel, that basically means the other monitors are not answering the probes required to establish the set of monitors to trigger an election
[17:03] <jluis> if you have 3 monitors, any given monitor needs to be able to reach at least 1 other monitor to get out of the probe state
[17:03] * EthanL (~lamberet@cce02cs4035-fa12-z.ams.hpecore.net) has joined #ceph
[17:03] <jluis> and by reach I mean being able to send a probe and getting a reply *before* the timeout expires
[17:04] * tallest_red (~djidis__@4MJAAHHJ3.tor-irc.dnsbl.oftc.net) Quit ()
[17:05] <vbel> jluis: thank you, do you know which timer would one increase? I have cluster of VMs, I guess I has slow guests and networking
[17:06] <jluis> you'll likely need to increase a few of them
[17:06] <jluis> can't recall them all and have to run
[17:06] <vbel> jluis: thanks!
[17:07] <jluis> but i think 'mon probe timeout', 'mon lease timeout' and some other may be the ones you want
[17:07] <jluis> check the src/common/config_opts.h and grep for 'mon.*timeout'
[17:07] * kutija (~kutija@89.216.27.139) Quit (Quit: Textual IRC Client: www.textualapp.com)
[17:08] <infernix> anyone aware of how mon_cluster_log_to_syslog and clog_to_syslog interact?
[17:08] * gauravbafna (~gauravbaf@122.178.243.68) Quit (Remote host closed the connection)
[17:12] <infernix> i think clog_to_syslog will let every individual component log directly to syslog and that includes mons
[17:13] <infernix> but mon_cluster_log_to_syslog seems to also send data to syslog in LogMonitor.cc, so is it just relaying clog data because "clog_to_monitors": "default=true"
[17:13] <infernix> ?
[17:14] * ntpttr_ (~ntpttr@134.134.139.74) Quit (Remote host closed the connection)
[17:14] * kawa2014 (~kawa@212.110.41.244) has joined #ceph
[17:19] * haomaiwang (~oftc-webi@114.249.211.190) has joined #ceph
[17:19] * sudocat (~dibarra@192.185.1.20) has joined #ceph
[17:20] * jdillaman (~jdillaman@pool-108-18-97-95.washdc.fios.verizon.net) has joined #ceph
[17:20] * Linkmark (~Linkmark@252.146-78-194.adsl-static.isp.belgacom.be) Quit (Quit: Leaving)
[17:27] * dyasny (~dyasny@192.222.152.136) Quit (Quit: Ex-Chat)
[17:32] * dyasny (~dyasny@cable-192.222.152.136.electronicbox.net) has joined #ceph
[17:32] * debian112 (~bcolbert@c-73-184-103-26.hsd1.ga.comcast.net) Quit (Ping timeout: 480 seconds)
[17:35] * rraja (~rraja@121.244.87.117) Quit (Quit: Leaving)
[17:40] * allaok (~allaok@machine107.orange-labs.com) Quit (Quit: Leaving.)
[17:40] * kawa2014 (~kawa@212.110.41.244) Quit (Ping timeout: 480 seconds)
[17:42] * huangjun|2 (~kvirc@117.152.65.191) Quit (Ping timeout: 480 seconds)
[17:42] * noah (~noah@eduroam-169-233-239-187.ucsc.edu) has joined #ceph
[17:43] * debian112 (~bcolbert@c-73-184-103-26.hsd1.ga.comcast.net) has joined #ceph
[17:43] * debian112 (~bcolbert@c-73-184-103-26.hsd1.ga.comcast.net) has left #ceph
[17:44] * ntpttr_ (~ntpttr@134.134.139.82) has joined #ceph
[17:44] * noah2 (~noah@eduroam-169-233-239-187.ucsc.edu) has joined #ceph
[17:45] * noah2 (~noah@eduroam-169-233-239-187.ucsc.edu) Quit ()
[17:45] * noah (~noah@eduroam-169-233-239-187.ucsc.edu) has left #ceph
[17:48] * noah (~noah@eduroam-169-233-239-187.ucsc.edu) has joined #ceph
[17:48] * Brochacho (~alberto@c-50-141-135-98.hsd1.il.comcast.net) has joined #ceph
[17:49] * noah is now known as noahw
[17:49] * kawa2014 (~kawa@89.184.114.246) has joined #ceph
[17:54] * mykola (~Mikolaj@193.93.217.54) has joined #ceph
[17:57] * penguinRaider (~KiKo@69.163.33.182) Quit (Ping timeout: 480 seconds)
[17:58] * ntpttr_ (~ntpttr@134.134.139.82) Quit (Quit: Leaving)
[17:59] * bvi (~Bastiaan@185.56.32.1) Quit (Quit: Leaving)
[18:01] * karnan (~karnan@106.51.138.205) has joined #ceph
[18:02] * derjohn_mob (~aj@fw.gkh-setu.de) Quit (Ping timeout: 480 seconds)
[18:03] * EthanL (~lamberet@cce02cs4035-fa12-z.ams.hpecore.net) Quit (Ping timeout: 480 seconds)
[18:06] * TMM (~hp@185.5.121.201) Quit (Quit: Ex-Chat)
[18:06] * penguinRaider (~KiKo@69.163.33.182) has joined #ceph
[18:07] * jidar (~jidar@r2d2.fap.me) has joined #ceph
[18:09] * linuxkidd (~linuxkidd@ip70-189-207-54.lv.lv.cox.net) has joined #ceph
[18:16] * gauravbafna (~gauravbaf@122.178.243.68) has joined #ceph
[18:17] <infernix> ok, worked out the syslog clog info
[18:17] <infernix> however, mons that have clog_to_syslog set are logging as 'unknown.0'
[18:18] <infernix> I'm not really sure why that is, because through the mon_cluster_log this isn't the case
[18:19] <infernix> only clog_to_syslog output does this
[18:19] * shubjero (~shubjero@107.155.107.246) has joined #ceph
[18:20] <infernix> that wouldn't be too bad since there is only one mon per host
[18:20] <infernix> but osds are doing the exact same thing
[18:20] <infernix> ceph-osd: unknown.0 :/0 0 : clog_to_syslog_facility = 'default=local0,audit=local1'
[18:20] * Be-El (~blinke@nat-router.computational.bio.uni-giessen.de) Quit (Quit: Leaving.)
[18:21] <infernix> seeing this for each OSD i enable clog_to_syslog for
[18:21] * bjornar_ (~bjornar@ti0099a430-0410.bb.online.no) Quit (Ping timeout: 480 seconds)
[18:22] * gauravbafna (~gauravbaf@122.178.243.68) Quit (Remote host closed the connection)
[18:22] * Skaag (~lunix@65.200.54.234) has joined #ceph
[18:22] * reed (~reed@142-254-29-216.dsl.dynamic.fusionbroadband.com) has joined #ceph
[18:29] * EthanL (~lamberet@cce02cs4035-fa12-z.ams.hpecore.net) has joined #ceph
[18:30] * derjohn_mob (~aj@88.128.80.62) has joined #ceph
[18:32] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit (Ping timeout: 480 seconds)
[18:34] * djidis__ (~qable@185.108.128.8) has joined #ceph
[18:35] * swami1 (~swami@27.7.169.81) has joined #ceph
[18:35] * joshd1 (~jdurgin@2602:30a:c089:2b0:3082:243e:bc2d:3f72) Quit (Quit: Leaving.)
[18:44] * fdmanana (~fdmanana@2001:8a0:6e0c:6601:9901:29b4:b6c5:3fa9) Quit (Ping timeout: 480 seconds)
[18:44] * kawa2014 (~kawa@89.184.114.246) Quit (Quit: Leaving)
[18:52] * dan__ (~Daniel@office.34sp.com) Quit (Quit: Leaving)
[18:53] * rinek (~o@62.109.134.112) Quit (Quit: ~)
[18:53] * rinek (~o@62.109.134.112) has joined #ceph
[18:54] * MrAbaddon (~MrAbaddon@184.99.136.95.rev.vodafone.pt) has joined #ceph
[18:55] * karnan (~karnan@106.51.138.205) Quit (Quit: Leaving)
[18:59] * joshd1 (~jdurgin@2602:30a:c089:2b0:3082:243e:bc2d:3f72) has joined #ceph
[19:00] * BrianA (~BrianA@fw-rw.shutterfly.com) has joined #ceph
[19:01] * gauravbafna (~gauravbaf@122.178.243.68) has joined #ceph
[19:02] * rwheeler (~rwheeler@nat-pool-bos-t.redhat.com) Quit (Quit: Leaving)
[19:04] * djidis__ (~qable@185.108.128.8) Quit ()
[19:04] <TheSov> does anyone know where ceph on ubuntu knows to mount the disks? i am trying to find a script or something, but i cannot seem to locate it
[19:06] * penguinRaider (~KiKo@69.163.33.182) Quit (Remote host closed the connection)
[19:06] * penguinRaider (~KiKo@69.163.33.182) has joined #ceph
[19:07] * sickology (~mio@vpn.bcs.hr) Quit (Read error: Connection reset by peer)
[19:07] * wjw-freebsd (~wjw@smtp.digiware.nl) has joined #ceph
[19:07] * BrianA1 (~BrianA@fw-rw.shutterfly.com) has joined #ceph
[19:09] * BrianA (~BrianA@fw-rw.shutterfly.com) Quit (Ping timeout: 480 seconds)
[19:13] * sickology (~mio@vpn.bcs.hr) has joined #ceph
[19:16] * post-factum (~post-fact@vulcan.natalenko.name) Quit (Killed (NickServ (Too many failed password attempts.)))
[19:16] * post-factum (~post-fact@vulcan.natalenko.name) has joined #ceph
[19:17] * gauravbafna (~gauravbaf@122.178.243.68) Quit (Remote host closed the connection)
[19:20] * ggarg is now known as ggarg_away
[19:21] * vasu (~vasu@c-73-231-60-138.hsd1.ca.comcast.net) has joined #ceph
[19:22] <infernix> ok, does anyone even use clog_to_syslog?
[19:23] <infernix> looks like in https://github.com/ceph/ceph/commit/7ef1273a519636aa008e84d828ad90d5cdaf5d46 someone commented out "+ //e.who = messenger->get_myinst();"
[19:23] <infernix> a few lines further down, e.log_to_syslog(get_log_prio(), get_syslog_facility());
[19:23] <infernix> so looks like that is where clog_to_syslog is logging data with "unknown.0"
[19:24] <infernix> because it isn't queued to LogClient like log_to_monitors is
[19:24] <infernix> does everyone just use clog_to_monitors and then dump monitor logs to syslog - instead of direct per process to syslog everywhere?
[19:25] * gauravbafna (~gauravbaf@122.178.243.68) has joined #ceph
[19:30] * gauravbafna (~gauravbaf@122.178.243.68) Quit (Read error: Connection reset by peer)
[19:31] * gauravbafna (~gauravbaf@122.178.243.68) has joined #ceph
[19:31] * gauravbafna (~gauravbaf@122.178.243.68) Quit (Remote host closed the connection)
[19:31] * stiopa (~stiopa@cpc73832-dals21-2-0-cust453.20-2.cable.virginm.net) has joined #ceph
[19:32] <georgem> TheSov: /lib/udev/rules.d/95-ceph-osd.rules
[19:36] <georgem> infernix: I'm logging by default to local files and the shipping the logs to ELK with logstash-forwarder
[19:38] * jermudgeon (~jhaustin@31.207.56.59) has joined #ceph
[19:38] * KindOne_ (~KindOne@h100.128.30.71.dynamic.ip.windstream.net) has joined #ceph
[19:39] * kefu (~kefu@114.92.118.31) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[19:40] * Aal (~Popz@46.166.186.215) has joined #ceph
[19:41] <TheSov> georgem, i dont get it, its correct in the udev rules yet it wont mount them on boot
[19:41] <TheSov> i ended up putting the osd mounts in fstab
[19:42] <georgem> TheSov: what versions of Ceph and Ubuntu?
[19:42] <TheSov> 10.2.2 and 16.04
[19:42] * Brochacho (~alberto@c-50-141-135-98.hsd1.il.comcast.net) Quit (Quit: Brochacho)
[19:43] <georgem> TheSov: check the mailing list, apparently the startup scripts are broken in the recent versions
[19:43] <TheSov> well that would make sense
[19:43] <ReSam> I'm having problems with rbdmap - it keeps failing with "Bad substitution". If I change the shebang to bash it works. is that a problem with my sh or should this be generally changed to use bash? https://github.com/ceph/ceph/blob/master/src/rbdmap#L1
[19:44] * BrianA (~BrianA@fw-rw.shutterfly.com) has joined #ceph
[19:44] * derjohn_mob (~aj@88.128.80.62) Quit (Ping timeout: 480 seconds)
[19:44] * niknakpaddywak (~xander.ni@23.227.28.5) Quit (Quit: Lost terminal)
[19:44] <georgem> TheSov: http://www.spinics.net/lists/ceph-users/msg28140.html
[19:45] * KindOne (~KindOne@0001a7db.user.oftc.net) Quit (Ping timeout: 480 seconds)
[19:45] * KindOne_ is now known as KindOne
[19:45] * BrianA1 (~BrianA@fw-rw.shutterfly.com) Quit (Ping timeout: 480 seconds)
[19:45] * niknakpaddywak (~xander.ni@outbound.lax.demandmedia.com) has joined #ceph
[19:45] <TheSov> ho damn
[19:46] * vikhyat (~vumrao@114.143.47.34) Quit (Quit: Leaving)
[19:47] * bjornar_ (~bjornar@ti0099a430-0410.bb.online.no) has joined #ceph
[19:49] * TMM (~hp@178-84-46-106.dynamic.upc.nl) has joined #ceph
[19:51] * rotbeard (~redbeard@2a02:908:df13:bb00:a863:41df:8783:1972) Quit (Quit: Leaving)
[19:53] * jermudgeon_ (~jhaustin@31.207.56.59) has joined #ceph
[19:53] * gauravbafna (~gauravbaf@122.178.243.68) has joined #ceph
[19:53] * jermudgeon (~jhaustin@31.207.56.59) Quit (Read error: No route to host)
[19:53] * jermudgeon_ is now known as jermudgeon
[19:54] * scg (~zscg@146-115-134-246.c3-0.nwt-ubr1.sbo-nwt.ma.cable.rcn.com) Quit (Remote host closed the connection)
[19:56] * scg (~zscg@146-115-134-246.c3-0.nwt-ubr1.sbo-nwt.ma.cable.rcn.com) has joined #ceph
[19:59] * ade (~abradshaw@dslb-188-102-071-182.188.102.pools.vodafone-ip.de) Quit (Ping timeout: 480 seconds)
[20:02] * treenerd_ (~gsulzberg@cpe90-146-148-47.liwest.at) has joined #ceph
[20:03] * newbie (~kvirc@host217-114-156-249.pppoe.mark-itt.net) has joined #ceph
[20:04] * jermudgeon (~jhaustin@31.207.56.59) Quit (Quit: jermudgeon)
[20:04] * Concubidated1 (~cube@139.26.13.160.rev.iijmobile.jp) Quit (Quit: Leaving.)
[20:04] * jermudgeon (~jhaustin@31.207.56.59) has joined #ceph
[20:04] * davidzlap (~Adium@2605:e000:1313:8003:65e2:27dc:6067:feb3) has joined #ceph
[20:06] * kazer (~kazer@2601:643:8104:ba60:eea8:6bff:fef8:449) has joined #ceph
[20:07] <infernix> Georgem, is it realtime?
[20:08] <georgem> infernix: logstash-forwarder watches the log file and ships the logs to ELK in real time
[20:09] <infernix> Still, clog to syslog seems bugged, if it wouldn't log everything as unknown.0 it would be usable
[20:09] <kazer> hi there
[20:09] <kazer> i'm having some issues with a new cluster i'm setting up: 2 peering, 429 creating+peering, 5 active+clean, 76 creating+incomplete
[20:09] <kazer> currently it's a 6 nodes, 18 osd cluster
[20:10] * Aal (~Popz@4MJAAHHP5.tor-irc.dnsbl.oftc.net) Quit ()
[20:10] <kazer> i can't get it to reach a sane state
[20:12] <kazer> i'm getting a bunch of 2016-07-06 14:06:30.822712 7f70cebac700 0 -- 192.168.21.20:6806/2879 >> 192.168.21.78:6804/1526 pipe(0x7f70ef945b80 sd=34 :47943 s=2 pgs=1619 cs=1 l=0 c=0x7f70efacdde0).fault, initiating reconnect in my OSD logs
[20:13] * rwheeler (~rwheeler@pool-173-48-195-215.bstnma.fios.verizon.net) has joined #ceph
[20:14] * n0x1d (~KUSmurf@aurora.enn.lu) has joined #ceph
[20:14] <kazer> tcpdump on 192.168.21.78 does show some traffic on port 6804, to and from the other nodes
[20:16] * jermudgeon (~jhaustin@31.207.56.59) Quit (Ping timeout: 480 seconds)
[20:19] * TMM (~hp@178-84-46-106.dynamic.upc.nl) Quit (Quit: Ex-Chat)
[20:22] * Jeffrey4l_ (~Jeffrey@110.244.236.63) Quit (Ping timeout: 480 seconds)
[20:22] * penguinRaider (~KiKo@69.163.33.182) Quit (Remote host closed the connection)
[20:23] * rakeshgm (~rakesh@106.51.28.105) has joined #ceph
[20:29] * treenerd_ (~gsulzberg@cpe90-146-148-47.liwest.at) Quit (Quit: treenerd_)
[20:29] * rendar (~I@host241-113-dynamic.51-82-r.retail.telecomitalia.it) Quit (Ping timeout: 480 seconds)
[20:29] * MentalRay (~MentalRay@office-mtl1-nat-146-218-70-69.gtcomm.net) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[20:30] * billwebb (~billwebb@50-203-47-138-static.hfc.comcastbusiness.net) Quit (Quit: billwebb)
[20:33] * dgurtner (~dgurtner@84-73-130-19.dclient.hispeed.ch) Quit (Ping timeout: 480 seconds)
[20:36] * penguinRaider (~KiKo@69.163.33.182) has joined #ceph
[20:42] * rakeshgm (~rakesh@106.51.28.105) Quit (Quit: Leaving)
[20:43] * MentalRay (~MentalRay@office-mtl1-nat-146-218-70-69.gtcomm.net) has joined #ceph
[20:44] * n0x1d (~KUSmurf@61TAAAAHM.tor-irc.dnsbl.oftc.net) Quit ()
[20:46] * mivaho (~quassel@2001:983:eeb4:1:c0de:69ff:fe2f:5599) Quit (Quit: No Ping reply in 180 seconds.)
[20:46] * mivaho (~quassel@2001:983:eeb4:1:c0de:69ff:fe2f:5599) has joined #ceph
[20:55] * rendar (~I@host241-113-dynamic.51-82-r.retail.telecomitalia.it) has joined #ceph
[20:56] * HappyLoaf (~HappyLoaf@cpc93928-bolt16-2-0-cust133.10-3.cable.virginm.net) has joined #ceph
[21:05] * Discovery (~Discovery@178.239.49.67) has joined #ceph
[21:15] * treenerd_ (~gsulzberg@cpe90-146-148-47.liwest.at) has joined #ceph
[21:16] * treenerd_ (~gsulzberg@cpe90-146-148-47.liwest.at) Quit ()
[21:21] * cronburg (~cronburg@nat-pool-bos-t.redhat.com) Quit (Ping timeout: 480 seconds)
[21:31] * cronburg (~cronburg@nat-pool-bos-t.redhat.com) has joined #ceph
[21:42] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[21:43] * derjohn_mob (~aj@x590e37e1.dyn.telefonica.de) has joined #ceph
[21:44] * Skaag (~lunix@65.200.54.234) Quit (Ping timeout: 480 seconds)
[21:48] * saintpablo (~saintpabl@185.85.5.78) has joined #ceph
[21:49] * swami1 (~swami@27.7.169.81) Quit (Quit: Leaving.)
[21:51] * saintpablo (~saintpabl@185.85.5.78) Quit ()
[21:52] * Skaag (~lunix@65.200.54.234) has joined #ceph
[22:05] * mykola (~Mikolaj@193.93.217.54) Quit (Quit: away)
[22:06] * gauravbafna (~gauravbaf@122.178.243.68) Quit (Remote host closed the connection)
[22:10] * joshd1 (~jdurgin@2602:30a:c089:2b0:3082:243e:bc2d:3f72) Quit (Quit: Leaving.)
[22:13] * georgem (~Adium@206.108.127.16) has left #ceph
[22:18] * mewald (~Adium@89.204.130.98) has joined #ceph
[22:20] <mewald> I am using https://github.com/rochaporto/collectd-ceph to monitor Ceph in Grafana. I see some "apply_latency_ms" values at around 300ms latency. Is that a usual value?
[22:21] * Discovery (~Discovery@178.239.49.67) Quit (Read error: Connection reset by peer)
[22:21] * bniver (~bniver@pool-173-48-58-27.bstnma.fios.verizon.net) Quit (Quit: Leaving)
[22:26] * Brochacho (~alberto@c-50-141-135-98.hsd1.il.comcast.net) has joined #ceph
[22:26] * moon (~moon@217-19-26-201.dsl.cambrium.nl) has joined #ceph
[22:32] * slowriot (~Maza@46.166.190.223) has joined #ceph
[22:42] * swami1 (~swami@27.7.169.81) has joined #ceph
[22:43] * mewald (~Adium@89.204.130.98) Quit (Quit: Leaving.)
[22:44] * derjohn_mobi (~aj@x590c58db.dyn.telefonica.de) has joined #ceph
[22:48] * derjohn_mob (~aj@x590e37e1.dyn.telefonica.de) Quit (Ping timeout: 480 seconds)
[22:55] * Brochacho (~alberto@c-50-141-135-98.hsd1.il.comcast.net) Quit (Quit: Brochacho)
[22:58] * gauravbafna (~gauravbaf@122.178.243.68) has joined #ceph
[23:02] * gauravbafna (~gauravbaf@122.178.243.68) Quit (Read error: Connection reset by peer)
[23:02] * slowriot (~Maza@46.166.190.223) Quit ()
[23:08] * wes_dillingham (~wes_dilli@140.247.242.44) Quit (Ping timeout: 480 seconds)
[23:16] * Brochacho (~alberto@c-50-141-135-98.hsd1.il.comcast.net) has joined #ceph
[23:20] * newbie (~kvirc@host217-114-156-249.pppoe.mark-itt.net) Quit (Ping timeout: 480 seconds)
[23:20] * mattbenjamin (~mbenjamin@12.118.3.106) Quit (Quit: Leaving.)
[23:23] * danieagle (~Daniel@201-95-102-169.dsl.telesp.net.br) Quit (Quit: Obrigado por Tudo! :-) inte+ :-))
[23:24] * BrianA (~BrianA@fw-rw.shutterfly.com) Quit (Read error: Connection reset by peer)
[23:29] * terminalecho (~terminale@vanquish.cacr.caltech.edu) Quit (Quit: -NO CARRIER-)
[23:39] * moon (~moon@217-19-26-201.dsl.cambrium.nl) Quit (Ping timeout: 480 seconds)
[23:44] * jclm (~jclm@ip68-96-196-245.lv.lv.cox.net) has joined #ceph
[23:44] * jclm (~jclm@ip68-96-196-245.lv.lv.cox.net) Quit ()
[23:47] * brad_mssw (~brad@66.129.88.50) Quit (Quit: Leaving)

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.