#ceph IRC Log

Index

IRC Log for 2016-02-24

Timestamps are in GMT/BST.

[0:00] * jclm (~jclm@63.117.50.130) has joined #ceph
[0:00] * jclm (~jclm@63.117.50.130) Quit ()
[0:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Remote host closed the connection)
[0:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[0:03] * brianjjo (~AG_Scott@76GAACOWK.tor-irc.dnsbl.oftc.net) Quit ()
[0:03] * Arcturus (~CobraKhan@kunstler.tor-exit.calyxinstitute.org) has joined #ceph
[0:04] * psiekl (psiekl@wombat.eu.org) has joined #ceph
[0:11] * davidzlap (~Adium@2605:e000:1313:8003:78b2:1b63:c4ce:b731) has joined #ceph
[0:11] * xarses (~xarses@64.124.158.100) Quit (Ping timeout: 480 seconds)
[0:11] * bjornar__ (~bjornar@ti0099a430-0908.bb.online.no) Quit (Ping timeout: 480 seconds)
[0:14] * davidzlap1 (~Adium@2605:e000:1313:8003:78b2:1b63:c4ce:b731) has joined #ceph
[0:14] * davidzlap (~Adium@2605:e000:1313:8003:78b2:1b63:c4ce:b731) Quit (Read error: Connection reset by peer)
[0:16] * mattbenjamin (~mbenjamin@sccc-66-78-236-243.smartcity.com) has joined #ceph
[0:19] * cathode (~cathode@50-198-166-81-static.hfc.comcastbusiness.net) Quit (Quit: Leaving)
[0:22] * MACscr|lappy (~MACscrlap@c-73-9-230-5.hsd1.il.comcast.net) Quit (Ping timeout: 480 seconds)
[0:24] * vbellur (~vijay@sccc-66-78-236-243.smartcity.com) Quit (Ping timeout: 480 seconds)
[0:25] * shtnik (~shtnik@37.49.162.23) has joined #ceph
[0:27] * neurodrone_ (~neurodron@pool-100-35-67-57.nwrknj.fios.verizon.net) Quit (Quit: neurodrone_)
[0:28] * xarses (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) has joined #ceph
[0:30] * vbellur (~vijay@sccc-66-78-236-243.smartcity.com) has joined #ceph
[0:33] * Arcturus (~CobraKhan@4MJAACUWV.tor-irc.dnsbl.oftc.net) Quit ()
[0:33] * lmg (~Jaska@tor00.telenet.unc.edu) has joined #ceph
[0:37] * reed_ (~reed@75-101-54-18.dsl.static.fusionbroadband.com) has joined #ceph
[0:37] * reed (~reed@75-101-54-18.dsl.static.fusionbroadband.com) Quit (Read error: Connection reset by peer)
[0:37] * davidzlap1 (~Adium@2605:e000:1313:8003:78b2:1b63:c4ce:b731) Quit (Read error: Connection reset by peer)
[0:37] * davidzlap (~Adium@2605:e000:1313:8003:78b2:1b63:c4ce:b731) has joined #ceph
[0:44] * davidzlap (~Adium@2605:e000:1313:8003:78b2:1b63:c4ce:b731) Quit (Read error: Connection reset by peer)
[0:44] * davidzlap (~Adium@2605:e000:1313:8003:78b2:1b63:c4ce:b731) has joined #ceph
[0:48] * shtnik (~shtnik@37.49.162.23) Quit (Ping timeout: 480 seconds)
[0:49] * vata (~vata@207.96.182.162) Quit (Ping timeout: 480 seconds)
[0:49] * haplo37 (~haplo37@199.91.185.156) Quit (Remote host closed the connection)
[0:51] * MentalRay (~MentalRay@MTRLPQ42-1176054809.sdsl.bell.ca) has joined #ceph
[0:53] * pvh_sa (~pvh@169-0-182-212.ip.afrihost.co.za) Quit (Ping timeout: 480 seconds)
[1:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Remote host closed the connection)
[1:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[1:03] * lmg (~Jaska@76GAACOYN.tor-irc.dnsbl.oftc.net) Quit ()
[1:03] * Kizzi (~Wijk@torsrva.snydernet.net) has joined #ceph
[1:08] * garphy`aw is now known as garphy
[1:10] * EinstCrazy (~EinstCraz@101.85.63.39) has joined #ceph
[1:13] * MentalRay (~MentalRay@MTRLPQ42-1176054809.sdsl.bell.ca) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[1:17] <motk> 10077 pgs peering
[1:17] <motk> 10077 pgs stuck inactive
[1:17] <motk> 10077 pgs stuck unclean
[1:17] <motk> awesome
[1:18] * EinstCrazy (~EinstCraz@101.85.63.39) Quit (Ping timeout: 480 seconds)
[1:19] * garphy is now known as garphy`aw
[1:20] * davidzlap1 (~Adium@cpe-172-91-154-245.socal.res.rr.com) has joined #ceph
[1:24] * wyang (~wyang@114.111.166.47) has joined #ceph
[1:25] <motk> I'll try an osd repair and see what happens
[1:26] * wyang (~wyang@114.111.166.47) Quit ()
[1:26] * davidzlap (~Adium@2605:e000:1313:8003:78b2:1b63:c4ce:b731) Quit (Ping timeout: 480 seconds)
[1:32] * Kioob (~Kioob@sal69-4-78-192-172-15.fbxo.proxad.net) Quit (Quit: Leaving.)
[1:33] * Kizzi (~Wijk@84ZAACT7P.tor-irc.dnsbl.oftc.net) Quit ()
[1:34] * Kioob1 (~Kioob@2a01:e34:ec0a:c0f0:250:b6ff:fe6a:2c02) has joined #ceph
[1:36] * vbellur (~vijay@sccc-66-78-236-243.smartcity.com) Quit (Ping timeout: 480 seconds)
[1:41] * davidzlap1 (~Adium@cpe-172-91-154-245.socal.res.rr.com) Quit (Read error: Connection reset by peer)
[1:43] * vata (~vata@cable-21.246.173-197.electronicbox.net) has joined #ceph
[1:47] * shtnik (~shtnik@188.120.252.178) has joined #ceph
[1:52] * sudocat1 (~dibarra@192.185.1.19) has joined #ceph
[1:53] * sudocat1 (~dibarra@192.185.1.19) Quit ()
[1:53] * sudocat1 (~dibarra@192.185.1.19) has joined #ceph
[1:53] * sudocat1 (~dibarra@192.185.1.19) Quit ()
[1:54] * mattbenjamin (~mbenjamin@sccc-66-78-236-243.smartcity.com) Quit (Ping timeout: 480 seconds)
[1:56] * MentalRay (~MentalRay@MTRLPQ42-1176054809.sdsl.bell.ca) has joined #ceph
[1:59] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[1:59] * sudocat (~dibarra@192.185.1.20) Quit (Ping timeout: 480 seconds)
[2:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Remote host closed the connection)
[2:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[2:03] * thansen (~thansen@17.253.sfcn.org) Quit (Quit: Ex-Chat)
[2:05] * branto (~branto@ip-78-102-208-28.net.upcbroadband.cz) Quit (Quit: Leaving.)
[2:06] <motk> anyone know how I can find out what's blokcing remapped+peering?
[2:07] * vbellur (~vijay@2601:647:4f00:4960:5e51:4fff:fee8:6a5c) has joined #ceph
[2:09] * LeaChim (~LeaChim@host86-171-90-242.range86-171.btcentralplus.com) Quit (Ping timeout: 480 seconds)
[2:11] * Racpatel (~Racpatel@2601:87:3:3601::2d54) Quit (Ping timeout: 480 seconds)
[2:16] * spudly (~spudly@zoot.murf.org) Quit (Quit: ZNC - http://znc.in)
[2:17] * johnhunter (~hunter@139.129.6.152) has joined #ceph
[2:18] <mischief> hey, any ceph devs around?
[2:18] * spudly (~spudly@zoot.murf.org) has joined #ceph
[2:19] * Racpatel (~Racpatel@2601:87:3:3601::2d54) has joined #ceph
[2:27] * spudly (~spudly@zoot.murf.org) Quit (Quit: ZNC - http://znc.in)
[2:29] * spudly (~spudly@zoot.murf.org) has joined #ceph
[2:30] * neerbeer (~neerbeer@2600:1003:b128:e57e:5876:2616:f9e8:f732) Quit (Ping timeout: 480 seconds)
[2:35] * Mika_c (~quassel@122.146.93.152) has joined #ceph
[2:36] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Remote host closed the connection)
[2:40] * yanzheng (~zhyan@182.139.22.205) has joined #ceph
[2:43] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[2:50] * Kupo (~tyler.wil@23.111.254.159) Quit (Quit: Leaving.)
[2:51] * fdmanana (~fdmanana@2001:8a0:6dfd:6d01:78b3:9839:3072:f158) Quit (Ping timeout: 480 seconds)
[3:05] * efirs (~firs@c-50-185-70-125.hsd1.ca.comcast.net) has joined #ceph
[3:05] * Kupo (~tyler.wil@23.111.254.159) has joined #ceph
[3:07] * joshd1 (~jdurgin@68-119-140-18.dhcp.ahvl.nc.charter.com) Quit (Quit: Leaving.)
[3:12] <motk> woo I made it worse
[3:12] <motk> 1 pgs down
[3:12] <motk> 23103 pgs peering
[3:12] <motk> 23103 pgs stuck inactive
[3:12] <motk> 23103 pgs stuck unclean
[3:14] <lurbs> If you have that many PGs is your ulimit high enough?
[3:16] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[3:17] * zhaochao (~zhaochao@cl-337.han-01.vn.sixxs.net) has joined #ceph
[3:18] <motk> and now a monitor is going down
[3:18] <motk> hrm, good question
[3:20] <motk> yep unlimited
[3:26] * neurodrone_ (~neurodron@162.243.191.67) has joined #ceph
[3:27] * kefu (~kefu@114.92.107.250) has joined #ceph
[3:28] * naoto (~naotok@27.131.11.254) has joined #ceph
[3:32] * zhaochao (~zhaochao@cl-337.han-01.vn.sixxs.net) Quit (Ping timeout: 480 seconds)
[3:40] * JohnO (~Guest1390@95.211.169.35) has joined #ceph
[3:42] * spudly (~spudly@zoot.murf.org) Quit (Quit: ZNC - http://znc.in)
[3:43] * zhaochao (~zhaochao@125.39.112.15) has joined #ceph
[3:43] * zhaochao (~zhaochao@125.39.112.15) Quit ()
[3:44] * spudly (~spudly@zoot.murf.org) has joined #ceph
[3:45] * spudly (~spudly@zoot.murf.org) Quit ()
[3:48] * MentalRay (~MentalRay@MTRLPQ42-1176054809.sdsl.bell.ca) Quit (Ping timeout: 480 seconds)
[3:48] * spudly (~spudly@zoot.murf.org) has joined #ceph
[3:49] * spudly (~spudly@zoot.murf.org) Quit ()
[3:52] * spudly (~spudly@zoot.murf.org) has joined #ceph
[3:53] * spudly (~spudly@zoot.murf.org) Quit ()
[3:54] * spudly (~spudly@zoot.murf.org) has joined #ceph
[3:55] * spudly (~spudly@zoot.murf.org) Quit ()
[3:59] * spudly (~spudly@zoot.murf.org) has joined #ceph
[4:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Remote host closed the connection)
[4:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[4:04] * bearkitten (~bearkitte@cpe-76-172-86-115.socal.res.rr.com) Quit (Quit: WeeChat 1.4)
[4:04] * spudly (~spudly@zoot.murf.org) Quit ()
[4:05] * spudly (~spudly@zoot.murf.org) has joined #ceph
[4:07] * spudly (~spudly@zoot.murf.org) Quit ()
[4:09] * bearkitten (~bearkitte@cpe-76-172-86-115.socal.res.rr.com) has joined #ceph
[4:09] * spudly (~spudly@zoot.murf.org) has joined #ceph
[4:10] * JohnO (~Guest1390@76GAACO3E.tor-irc.dnsbl.oftc.net) Quit ()
[4:11] * hgjhgjh (~pepzi@tor-relay.zwiebeltoralf.de) has joined #ceph
[4:12] * Kioob1 (~Kioob@2a01:e34:ec0a:c0f0:250:b6ff:fe6a:2c02) Quit (Ping timeout: 480 seconds)
[4:12] * shaunm (~shaunm@74.83.215.100) has joined #ceph
[4:13] * spudly (~spudly@zoot.murf.org) Quit ()
[4:14] * vasu (~vasu@c-73-231-60-138.hsd1.ca.comcast.net) Quit (Quit: Leaving)
[4:14] * spudly (~spudly@zoot.murf.org) has joined #ceph
[4:15] * johnhunter (~hunter@139.129.6.152) Quit (Read error: Connection reset by peer)
[4:15] * spudly (~spudly@zoot.murf.org) Quit ()
[4:17] * spudly (~spudly@zoot.murf.org) has joined #ceph
[4:18] * neurodrone_ (~neurodron@162.243.191.67) Quit (Ping timeout: 480 seconds)
[4:20] * spudly (~spudly@zoot.murf.org) Quit ()
[4:20] * skullone (~skullone@shell.skull-tech.com) Quit (Read error: Connection reset by peer)
[4:20] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Remote host closed the connection)
[4:21] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[4:22] * spudly (~spudly@zoot.murf.org) has joined #ceph
[4:23] * valeech (~valeech@pool-108-44-162-111.clppva.fios.verizon.net) Quit (Quit: valeech)
[4:24] * neurodrone_ (~neurodron@162.243.191.67) has joined #ceph
[4:26] <ben3> ssd selection with ceph seems to be a big deal.
[4:26] <ben3> am i right in thinking that the only drives with high endurance are really expensive
[4:26] <ben3> and do people usually use lower endurance drives with more of them and just swap them out when they fail?
[4:29] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Ping timeout: 480 seconds)
[4:40] * spudly (~spudly@zoot.murf.org) Quit (Quit: ZNC - http://znc.in)
[4:40] * hgjhgjh (~pepzi@84ZAACUB7.tor-irc.dnsbl.oftc.net) Quit ()
[4:41] * eXeler0n (~Szernex@torproxy02.31173.se) has joined #ceph
[4:43] * vikhyat (~vumrao@114.143.41.1) has joined #ceph
[4:44] * spudly (~spudly@zoot.murf.org) has joined #ceph
[4:51] * cooldharma06 (~chatzilla@14.139.180.40) Quit (Quit: ChatZilla 0.9.92 [Iceweasel 21.0/20130515140136])
[4:52] * linuxkidd (~linuxkidd@170.sub-70-193-125.myvzw.com) Quit (Remote host closed the connection)
[4:54] * swami1 (~swami@49.38.0.28) has joined #ceph
[4:56] * overclk (~vshankar@121.244.87.117) has joined #ceph
[5:01] * pabluk__ (~pabluk@2a01:e34:ec16:93e0:fd88:f6b3:9123:7990) Quit (Ping timeout: 480 seconds)
[5:05] * spudly (~spudly@zoot.murf.org) has left #ceph
[5:07] * MentalRay (~MentalRay@107.171.161.165) has joined #ceph
[5:07] * pvh_sa (~pvh@169-0-182-212.ip.afrihost.co.za) has joined #ceph
[5:08] * neurodrone_ (~neurodron@162.243.191.67) Quit (Quit: neurodrone_)
[5:09] * pabluk__ (~pabluk@2a01:e34:ec16:93e0:3d95:537c:fe:9b6c) has joined #ceph
[5:09] * shaunm (~shaunm@74.83.215.100) Quit (Ping timeout: 480 seconds)
[5:10] * eXeler0n (~Szernex@84ZAACUCW.tor-irc.dnsbl.oftc.net) Quit ()
[5:10] * Pommesgabel (~lmg@tor.yrk.urgs.uk0.bigv.io) has joined #ceph
[5:17] * kanagaraj (~kanagaraj@121.244.87.117) has joined #ceph
[5:22] * Vacuum__ (~Vacuum@88.130.221.85) has joined #ceph
[5:28] * ereb0s (~ereb0s@107-217-135-120.lightspeed.lsvlky.sbcglobal.net) Quit (Ping timeout: 480 seconds)
[5:28] * Vacuum_ (~Vacuum@i59F7978F.versanet.de) Quit (Ping timeout: 480 seconds)
[5:38] * wjw-freebsd (~wjw@smtp.digiware.nl) Quit (Ping timeout: 480 seconds)
[5:38] * branto (~branto@ip-78-102-208-28.net.upcbroadband.cz) has joined #ceph
[5:40] * DV (~veillard@2001:41d0:a:f29f::1) Quit (Remote host closed the connection)
[5:40] * DV (~veillard@2001:41d0:a:f29f::1) has joined #ceph
[5:40] * Pommesgabel (~lmg@84ZAACUDO.tor-irc.dnsbl.oftc.net) Quit ()
[5:40] * Joppe4899 (~Miho@192.42.115.101) has joined #ceph
[5:47] * shylesh (~shylesh@121.244.87.118) has joined #ceph
[5:47] * rdas (~rdas@121.244.87.116) has joined #ceph
[5:50] * _NiC (~kristian@aeryn.ronningen.no) Quit (Ping timeout: 480 seconds)
[5:50] * _NiC (~kristian@aeryn.ronningen.no) has joined #ceph
[6:01] * karnan (~karnan@121.244.87.117) has joined #ceph
[6:04] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[6:05] * jamespage (~jamespage@2a00:1098:0:80:1000:42:0:1) Quit (Quit: Coyote finally caught me)
[6:06] * jamespage (~jamespage@2a00:1098:0:80:1000:42:0:1) has joined #ceph
[6:10] * Joppe4899 (~Miho@76GAACO52.tor-irc.dnsbl.oftc.net) Quit ()
[6:11] * CydeWeys (~CoMa@93.115.95.207) has joined #ceph
[6:18] <iggy> ben3: that's pretty standard for SSDs yeah (not just with ceph)
[6:18] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit (Ping timeout: 480 seconds)
[6:19] <ben3> ceph does all writes to journal though
[6:19] <ben3> so increases the amount of writes
[6:19] <iggy> either buy better up front or just accept that you are going to replace pretty often... depends on a lot of different things which way is better in the long run
[6:19] <ben3> yeah it's hard to know which way to go
[6:19] <iggy> we do a lot of writes... for us going cheap and accepting that we'll replace often was better long term
[6:20] <ben3> it'd be a lot easier if bulk sequential writes could bypass journal
[6:20] <iggy> use different pools?
[6:20] * vikhyat (~vumrao@114.143.41.1) Quit (Quit: Leaving)
[6:20] <ben3> well most drives seem to be rated at 100 to 200tb writes?
[6:20] <ben3> i'm not really sure what kind of write load is common
[6:21] <ben3> but i'm guessing that is about what you'd get out of 3 or 4 hard-disks?
[6:21] <ben3> if that.
[6:21] <iggy> I've considered multiple pools where one is pure SSD, one is SSD journals/spinners, and one is all spinners
[6:22] <iggy> but if you don't have a clear picture of what your workload is going to be, it's hard to tell tell everything up front where to write to
[6:23] <ben3> well if it's 4k writes go to ssd, 4mb go to disk?
[6:23] <ben3> with some cut off in between
[6:24] * pvh_sa (~pvh@169-0-182-212.ip.afrihost.co.za) Quit (Ping timeout: 480 seconds)
[6:25] <ben3> i dunno with zfs/zil the writes going to journal is much more managable
[6:25] <ben3> i'm not sure if layering on top would work, or if it's even a good idea
[6:25] <iggy> but like say something like VPS hosting (think digital ocean/linode/vultr/etc) was your target workload... you could have your rootfs' go to SSD, larger volumes go to hybrid, snapshots go to spinners
[6:25] <iggy> or something
[6:25] <m0zes> the problem with "unreliable" ssds as journals is that there is very good chance it takes multiple osds out with a single ssd failure.
[6:26] <iggy> 10k servers... you don't really miss one
[6:26] <iggy> but yeah, that is something to consider
[6:26] <ben3> yeah it's using ceph on smaller scale that is more complicated :)
[6:26] <ben3> like cache tiering doesn't sound to me like it'd give a lot of benefit
[6:27] <m0zes> it doesn't really, imho...
[6:27] <m0zes> speaking as someone who is currently using it.
[6:27] <ben3> heh
[6:27] <ben3> i didn't find zfs l2arc helping much
[6:28] <m0zes> bluestore!
[6:28] * flisky (~Thunderbi@36.110.40.28) has joined #ceph
[6:28] <ben3> well it did for metadata a bit
[6:28] <ben3> but not for normal data
[6:28] <ben3> because you can end up getting some blocks in / some out
[6:28] <ben3> taking slight read load off
[6:28] <ben3> but writes are much more expensive than reads anyway
[6:29] <m0zes> I've got my cephfs metadata on *nice* ssd. I wish the mds server wasn't quite as single threaded as it feels.
[6:29] <ben3> ahh
[6:29] <ben3> isn't cephfs beta still?
[6:29] <ben3> i am mostly looking at rados
[6:29] <ben3> and yeah vm storage
[6:29] <iggy> I probably wouldn't use ceph on a small scale
[6:29] <ben3> so 20/80 read/write load or thereabouts
[6:29] <m0zes> they aren't *really* recommending production workloads for it. sometimes you have to live on the edge.
[6:30] <ben3> iggy why is that?
[6:30] <ben3> because you want at least 3 way replication?
[6:30] <m0zes> especially considering glusterfs metadata is horrendous when it comes to *stupid* users with 1M+ files in single directories
[6:30] <m0zes> all <4KB
[6:31] <ben3> heh i haven't used many distributed file systems, at first i thought it'd be nifty..
[6:31] <ben3> but nothing seemed to do nice read biases for multiple locations
[6:31] <ben3> so my whole idea of storing data and replicating in multiple locations meant high write latency, slow metadata access etc
[6:31] <m0zes> it is really nifty, but geolocation is a *really* hard problem to solve.
[6:31] <ben3> and then reads coming from somewhere far away rather than close
[6:32] * adun153 (~ljtirazon@112.198.83.216) has joined #ceph
[6:32] <m0zes> unless your sites are accessing different subsets of data.
[6:32] <ben3> well yeah you'd usually access subsets of data
[6:32] <ben3> but you want to be able to access from other locations too
[6:32] <iggy> ben3: I just think there are better options on a small scale
[6:33] <m0zes> then cephfs. subdirs going to different pools. each with site locality...
[6:33] <ben3> iggy: like?
[6:33] <ben3> m0zes: that's not a bad idea
[6:34] <m0zes> primary on the the local set of disks. secondaries at other sites. reads come local. writes have to hit all, but that happens...
[6:34] <lincolnb> the "user with 1M 4KB files in a single dir" use case is very common in scientific/cluster computing i find :)
[6:34] <m0zes> you could even do something like a local cachetier for the local site. reads/writes hit local, then get flushed out.
[6:35] <ben3> m0zes: now that's getting interesting
[6:35] <m0zes> lincolnb: indeed. and it doesn't matter how many times I tell them it is a bad idea...
[6:35] * vikhyat (~vumrao@121.244.87.116) has joined #ceph
[6:35] <ben3> i wonder if multitiering in general is a good idea
[6:35] <ben3> and having close storage that is fast
[6:35] <lincolnb> m0zes: honestly the best thing ive ever done is remove the alias of 'ls' to 'ls --color'
[6:35] <ben3> and remote storage that is huge
[6:35] <lincolnb> theres an enormous number of stats with colorful ls
[6:36] <ben3> lincolnb: god i hate colorls.
[6:36] <ben3> didn't redhat do that by default?
[6:36] <lincolnb> yes.
[6:36] <lincolnb> we use EL6 and it is the default
[6:36] <ben3> for some reason i always seem to use ls -ltar
[6:36] <ben3> it's like programmed into me... even when i don't need it ..
[6:36] <lincolnb> fortunately easily removed by just deleting whatever file in /etc/profile.d IIRC
[6:37] <ben3> you should chsh them while you're at it :)
[6:37] <iggy> if you want to do multi-site, can't you tune to have 3 copies, but ack when there are 2? so you have 2 local copies, the ack goes to the writer, and the 3rd (remote) write happens effectively asynchronously
[6:37] <lincolnb> :)
[6:37] <ben3> iggy: oh can you? that'd be nice in general
[6:37] <m0zes> iggy: I was under the impression that all osds that were up+in had to ack... even for min_size < size.
[6:38] <ben3> am i the only crazy person who likes stracing binaries to see how many system calls they make?
[6:38] <lincolnb> no
[6:38] <ben3> have you tried stracing curl?
[6:38] <m0zes> ben3: no, but we are few and far between
[6:38] <iggy> m0zes: min_size was what I was thinking, and I could very well be wrong about how it works
[6:38] * adun153 (~ljtirazon@112.198.83.216) Quit (Read error: Connection reset by peer)
[6:38] <ben3> curl writes to the screen one character at a time
[6:38] <lincolnb> huh.
[6:38] <iggy> so my suggestion may be wishful thinking
[6:39] * adun153 (~ljtirazon@112.198.83.216) has joined #ceph
[6:39] <ben3> but yeah lots of programs do heaps of system calls and they're all synchronous and sequential
[6:40] <ben3> and so you kind of need low access latency times to get high performance
[6:40] <ben3> where often what one is doing doesn't really require it; it knows ahead what it needs, there's just no easy/nice way to fire off lots of requests at once
[6:40] * CydeWeys (~CoMa@4MJAACU8K.tor-irc.dnsbl.oftc.net) Quit ()
[6:41] * Heliwr (~w2k@84ZAACUFZ.tor-irc.dnsbl.oftc.net) has joined #ceph
[6:41] <m0zes> curls behavior is still better than the grad student I had that wrote a custom java (de)serializer that wrote data structures out as comma seperated binary values. 0,1,0,1,1,1,0...
[6:41] <ben3> m0zes: haha
[6:41] <ben3> i tried writing my own minimal curl implementation
[6:41] <ben3> i was trying to see how fast i could make it
[6:41] <ben3> but i couldn't figure out any nice way to pass reference and receive data and write to file system
[6:41] <ben3> without read/write loop
[6:42] <ben3> but i managed to get performance faster and faster.. then i changed to using a different cpu, and i got way more improvement
[6:42] <ben3> then i lost interest :)
[6:43] <m0zes> the other component of curl is that the time for a syscall to print is < the latency of the connection. it doesn't hurt it to print 1 char at a time, even if it is just for the illusion that the connection is working.
[6:43] <ben3> but like just downloading using curl on core2duo 3ghz will use 10 to 20% cpu depending on interrupt mitigation.
[6:43] * adun153 (~ljtirazon@112.198.83.216) Quit (Read error: Connection reset by peer)
[6:43] <ben3> and over infiniband you can easily use 50%+ cpu on i7-3770
[6:43] <ben3> err gigabit on core2duo
[6:43] * adun153 (~ljtirazon@112.198.83.216) has joined #ceph
[6:44] <ben3> but then you look at things like adsl/vdsl and the cpu usage is actually much higher
[6:44] <ben3> well compared to the bandwidth. the connection is slow enough that colaescing can't work, so you end up with 1460 bytes here, 1460 bytes there
[6:44] <m0zes> so, don't curl over infiniband. there are much better applications/protocols for transferring binary data between machines. http is just used for the web because it is supported by all.
[6:45] <ben3> true, i was curious partially
[6:45] <ben3> i got started when i ldd'ed curl
[6:45] <ben3> yeh rdma is the way to go for infiniband
[6:46] <m0zes> and I'm sure the reasons for a printf on every character is because some server long ago didn't behave properly unless the buffer was constantly polled.
[6:46] <ben3> also i wanted to add tcp fast open to curl and was struggling with the code a little
[6:47] <ben3> tcp fast open means you need to put the tcp open connection and initial packet together
[6:47] <ben3> for a 1 rtt saving
[6:47] <m0zes> remember the old philosophy. be strict with what you send, relaxed with what you recieve.
[6:47] <ben3> i'm surprised tcp fast open hasn't taken off more
[6:48] <ben3> it's safer than ecn..
[6:48] <m0zes> I enable it everywhere, but considering it is technically breaking the standard...
[6:49] * kefu (~kefu@114.92.107.250) Quit (Max SendQ exceeded)
[6:49] <ben3> it can fallback
[6:49] <ben3> reducing the initial retransmit time "breaks" the standard too
[6:49] <ben3> but 3+ seconds is crazy on the modern internet
[6:49] * kefu (~kefu@114.92.107.250) has joined #ceph
[6:50] <ben3> so linux defaulting to 1 second is a welcome change
[6:50] * m0zes has a 40Gb pipe. I know all about silly things breaking the internet ;)
[6:52] <ben3> you have 40 gigabit internet?
[6:52] <ben3> or 40 gigabit lan?
[6:52] <ben3> i've got 32gigabit infiniband at home heh
[6:52] <ben3> it's so cheap now!
[6:54] <m0zes> 40Gb to the campus core. the campus *stupidly* limits their 10Gb connection to the internet to 4Gb (due to stupid firewall per-gb pricing scheme). I'm getting a 100Gb *seperate* pipe to I2 "soon".
[6:54] <ben3> i wouldn't mind 40 gigabit internet :)
[6:54] <ben3> but 40gbe cards are expenive
[6:55] <lincolnb> Nx10Gb LAGs can be tricky
[6:55] <m0zes> 40GbE is cheap. so are switches. ~$400 isn't too bad http://www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct=1326
[6:56] <lincolnb> we have an 8x10Gb LAG. can be tough to push that much through heh.
[6:56] * linjan__ (~linjan@176.195.185.217) has joined #ceph
[6:57] <ben3> i'm using connectx-2s
[6:57] * m0zes has 5x 32-port 40Gb switches, ~$12,000 each. looking for some 100Gb ones soon. of course nothing can drive more than ~80Gb at the moment...
[6:57] <ben3> i can only get ~26 gigabit
[6:57] <lincolnb> what are you using for your ceph backend m0zes ?
[6:57] <m0zes> by "nothing" I mean no single computer...
[6:58] <ben3> 32 port 40gb nice
[6:58] <ben3> so you're using 40gbe in preference to infiniband?
[6:58] <m0zes> lincolnb: 24x Dell r730xd machines. each with 12x6TB drives, 4x 4TB drives, 2 400GB intel P3700 ssds, and 2 *cheap* ssds for os.
[6:59] <lincolnb> nice, similar here.
[6:59] <ben3> m0zes: so p3700 can keep up with 16 drives ok?
[6:59] <ben3> or it doesn't matter?
[7:00] <lincolnb> have ~30 r730xds with 6TB and some 8TB, different generations
[7:00] <m0zes> ben3: we've got a mix. 40GbE with RoCE support is slightly better than QDR IB. and it is ethernet, which we have more applications that can utilize.
[7:00] <lincolnb> iirc you are using EC w/ CephFS?
[7:00] <ben3> ahh
[7:00] <m0zes> ben3: 8x drives off each P3700.
[7:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Remote host closed the connection)
[7:01] * treenerd_ (~gsulzberg@cpe90-146-148-47.liwest.at) has joined #ceph
[7:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[7:01] <m0zes> lincolnb: yep. and I'm hating it for most things.
[7:01] <lincolnb> :/
[7:01] <lincolnb> do you have problems w/ blocked ops?
[7:01] <lincolnb> or why do you hate it?
[7:04] <m0zes> not really the blocked ops. just slow performance. we've got homedirs on a cache-tier+EC. we'll be re-architecting (again) moving homedirs to a (quota'd) replicated pool, have a scratch pool, and a "large files" pool that will be EC backed.
[7:05] * TomasCZ (~TomasCZ@yes.tenlab.net) Quit (Quit: Leaving)
[7:05] <m0zes> I'd prefer to make "large files" more of an archival pool and only available on the headnodes. force users to copy off the files they are working on to scratch. the boss thinks that will be too hard for users.
[7:05] <lincolnb> gotcha. yeah, we've got homedirs on an NFS and a large "scratch" on our EC pool
[7:06] <lincolnb> it works OK, but i tend to get a lot of blocked ops when usage gets heavy
[7:06] * lincolnb shrugs
[7:06] <m0zes> lincolnb: what version of ceph?
[7:06] <lincolnb> 0.94.5
[7:06] <lincolnb> ssd journals, cache tier and ec pool on the same disks
[7:07] <m0zes> beware the move to infernalis. it changes tiering characteristics. reads don't force a migration from the underlying tier to the cache. thereby making it harder/slower unless you *really* have the tiering setup perfectly.
[7:08] <lincolnb> hm, good to know.
[7:09] <m0zes> and tiering+ecpool on the same disks caused blocked ops for us too.
[7:10] <m0zes> my solution was to take the 4TB disks as the cache tier, and the 6TB ones as the ec pool.
[7:10] <lincolnb> nice, yeah. i figured i'd take a few disks from each box and try that at some point.
[7:10] * Heliwr (~w2k@84ZAACUFZ.tor-irc.dnsbl.oftc.net) Quit ()
[7:11] * adun153 (~ljtirazon@112.198.83.216) Quit (Ping timeout: 480 seconds)
[7:11] * TomyLobo (~bret@91.109.29.120) has joined #ceph
[7:11] <ben3> so tiering with sata disks helps vs ec coding enough that you don't worry about ssd?
[7:11] <m0zes> seperating the duties of the disks should help quite a bit.
[7:11] <lincolnb> wasn't sure how straight forward it would be to drain the cache tier, detach, and attach a new one backed by a separate pool of disks
[7:11] <lincolnb> assuming thats what id need to do
[7:12] <ben3> oh right just because less load on the dsiks
[7:12] <ben3> disks
[7:12] <m0zes> lincolnb: I created a couple new crush rulesets and changed the pools to use them
[7:12] <m0zes> letting it migrate the existing pools live.
[7:13] <lincolnb> cool, that's great to know.
[7:13] * linjan_ (~linjan@176.195.241.98) has joined #ceph
[7:13] * DLange (~DLange@dlange.user.oftc.net) Quit (Quit: irssi messed up, cleaning the crufty monster)
[7:13] <m0zes> I'd do the ec pool first (mostly because it will take longer)
[7:14] <ben3> does having extra ram on osd nodes help much for read performance?
[7:14] <m0zes> as with most things, it depends on the workload :)
[7:14] <ben3> i suppose that could end up being like a ssd
[7:15] <ben3> with 24 boxes... there's sure to be quite a lot of ram available for disk cache
[7:15] <m0zes> ceph will use the page cache. doesn't necessarily mean that most of your reads are recently read or written.
[7:17] <m0zes> if anyone is interested in *crazy* things https://bitbucket.org/nikratio/s3ql/
[7:17] * shohn (~shohn@220.Red-79-146-26.dynamicIP.rima-tde.net) has joined #ceph
[7:17] * m0zes is using that as a backing store for os mirrors at the moment. still beta testing it off ceph radosgw...
[7:18] * DLange (~DLange@dlange.user.oftc.net) has joined #ceph
[7:18] <ben3> i agree that it's crazy :)
[7:20] <m0zes> anyway. it is getting late here, and I've got work in the morning. good night all
[7:20] * linjan__ (~linjan@176.195.185.217) Quit (Ping timeout: 480 seconds)
[7:21] * adun153 (~ljtirazon@112.198.83.216) has joined #ceph
[7:21] <lincolnb> later
[7:21] * linjan_ (~linjan@176.195.241.98) Quit (Ping timeout: 480 seconds)
[7:24] * Geph (~Geoffrey@169-0-138-51.ip.afrihost.co.za) Quit (Ping timeout: 480 seconds)
[7:30] * MentalRay (~MentalRay@107.171.161.165) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[7:32] * zaitcev (~zaitcev@c-50-130-189-82.hsd1.nm.comcast.net) Quit (Ping timeout: 480 seconds)
[7:35] * shyu (~Shanzhi@119.254.120.66) has joined #ceph
[7:40] * branto (~branto@ip-78-102-208-28.net.upcbroadband.cz) Quit (Quit: Leaving.)
[7:40] * derjohn_mobi (~aj@x590dfb93.dyn.telefonica.de) has joined #ceph
[7:41] * TomyLobo (~bret@4MJAACU98.tor-irc.dnsbl.oftc.net) Quit ()
[7:42] <hoonetorg> iggy: you ceph user??? and you didn't write a really fancy salt-ceph-formula yet as salt specialist???
[7:42] <hoonetorg> see u 1st time here!!!
[7:42] <hoonetorg> :)
[7:43] <iggy> we have one, I didn't write it... it is very intertwined with our salt setup here
[7:44] <hoonetorg> is it public??
[7:44] <hoonetorg> can hv look ???
[7:44] <iggy> nope
[7:44] <hoonetorg> :(
[7:44] <iggy> unfortunately
[7:45] <hoonetorg> I hoped
[7:45] * rf`1 (~skney@tor-exit2.15-cloud.fr) has joined #ceph
[7:45] * rakeshgm (~rakesh@106.51.225.55) has joined #ceph
[7:45] <hoonetorg> would be nice if someday a good formula exists
[7:46] <iggy> I suspect anything would be pretty tightly coupled to a setup
[7:46] <hoonetorg> the ansible guys have a nice one from sebastian han
[7:47] <hoonetorg> but me not using ansible :)
[7:48] <hoonetorg> iggy: hv you seen the request from oms101 regarding a sesceph module much ^^^ here?
[7:48] * Mika_c (~quassel@122.146.93.152) Quit (Remote host closed the connection)
[7:49] <hoonetorg> he's writing a module to manage ceph stuff
[7:49] <iggy> nice
[7:49] * shyu (~Shanzhi@119.254.120.66) Quit (Remote host closed the connection)
[7:50] <hoonetorg> i told him to ask salt guys early if he want's to include upstream, don't know if he did
[7:50] * shyu (~Shanzhi@119.254.120.66) has joined #ceph
[7:54] * Be-El (~blinke@nat-router.computational.bio.uni-giessen.de) has joined #ceph
[7:57] * kefu is now known as kefu|afk
[7:59] * kefu|afk is now known as kefu
[8:00] <iggy> I wouldn't know either, but if he has an issue open for it, an @iggy on it wouldn't hurt
[8:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Remote host closed the connection)
[8:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[8:15] * rf`1 (~skney@7V7AACU32.tor-irc.dnsbl.oftc.net) Quit ()
[8:15] * Eman1 (~datagutt@4MJAACVB0.tor-irc.dnsbl.oftc.net) has joined #ceph
[8:16] * adun153 (~ljtirazon@112.198.83.216) Quit (Ping timeout: 480 seconds)
[8:17] * flisky (~Thunderbi@36.110.40.28) Quit (Ping timeout: 480 seconds)
[8:18] * fdmanana (~fdmanana@2001:8a0:6dfd:6d01:4963:611d:d4e9:c457) has joined #ceph
[8:18] * Kioob1 (~Kioob@2a01:e34:ec0a:c0f0:250:b6ff:fe6a:2c02) has joined #ceph
[8:18] * zhaochao (~zhaochao@125.39.112.15) has joined #ceph
[8:19] * Kioob1 (~Kioob@2a01:e34:ec0a:c0f0:250:b6ff:fe6a:2c02) Quit ()
[8:20] * Kioob1 (~Kioob@2a01:e34:ec0a:c0f0:250:b6ff:fe6a:2c02) has joined #ceph
[8:22] * pvh_sa (~pvh@6c-88-14-d6-a1-4c.lan.uct.ac.za) has joined #ceph
[8:29] * owlbot (~supybot@pct-empresas-50.uc3m.es) Quit (Ping timeout: 480 seconds)
[8:29] * owlbot (~supybot@pct-empresas-50.uc3m.es) has joined #ceph
[8:29] * Kioob1 (~Kioob@2a01:e34:ec0a:c0f0:250:b6ff:fe6a:2c02) Quit (Ping timeout: 480 seconds)
[8:29] * branto (~branto@nat-pool-brq-t.redhat.com) has joined #ceph
[8:30] * pvh_sa (~pvh@6c-88-14-d6-a1-4c.lan.uct.ac.za) Quit (Ping timeout: 480 seconds)
[8:31] * pvh_sa (~pvh@6c-88-14-d6-a1-4c.lan.uct.ac.za) has joined #ceph
[8:39] * pvh_sa (~pvh@6c-88-14-d6-a1-4c.lan.uct.ac.za) Quit (Ping timeout: 480 seconds)
[8:44] * vata (~vata@cable-21.246.173-197.electronicbox.net) Quit (Quit: Leaving.)
[8:45] * Eman1 (~datagutt@4MJAACVB0.tor-irc.dnsbl.oftc.net) Quit ()
[8:45] * Dragonshadow (~AG_Scott@76GAACO9T.tor-irc.dnsbl.oftc.net) has joined #ceph
[8:46] * shohn (~shohn@220.Red-79-146-26.dynamicIP.rima-tde.net) Quit (Ping timeout: 480 seconds)
[8:47] * rakeshgm (~rakesh@106.51.225.55) Quit (Quit: Leaving)
[8:47] * enax (~enax@hq.ezit.hu) has joined #ceph
[8:47] * pvh_sa (~pvh@137.158.201.206) has joined #ceph
[8:47] * dersini (~chio@cpe-65-30-54-170.wi.res.rr.com) Quit (Ping timeout: 480 seconds)
[8:51] * adun153 (~ljtirazon@112.198.101.249) has joined #ceph
[8:52] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Remote host closed the connection)
[8:53] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[8:53] * linjan_ (~linjan@86.62.112.22) has joined #ceph
[8:53] * derjohn_mobi (~aj@x590dfb93.dyn.telefonica.de) Quit (Ping timeout: 480 seconds)
[8:55] * pvh_sa (~pvh@137.158.201.206) Quit (Ping timeout: 480 seconds)
[8:56] * thomnico (~thomnico@95.215.123.58) has joined #ceph
[9:00] * T1w (~jens@node3.survey-it.dk) has joined #ceph
[9:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Remote host closed the connection)
[9:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[9:03] * pvh_sa (~pvh@137.158.201.206) has joined #ceph
[9:05] * oms101 (~oms101@p20030057EA053A00C6D987FFFE4339A1.dip0.t-ipconnect.de) has joined #ceph
[9:10] * thomnico (~thomnico@95.215.123.58) Quit (Ping timeout: 480 seconds)
[9:11] * pvh_sa (~pvh@137.158.201.206) Quit (Ping timeout: 480 seconds)
[9:11] * rakeshgm (~rakesh@106.51.225.55) has joined #ceph
[9:12] * yanzheng1 (~zhyan@182.139.20.43) has joined #ceph
[9:12] * daviddcc (~dcasier@84.197.151.77.rev.sfr.net) Quit (Ping timeout: 480 seconds)
[9:13] * nardial (~ls@dslb-178-011-177-180.178.011.pools.vodafone-ip.de) has joined #ceph
[9:15] * Dragonshadow (~AG_Scott@76GAACO9T.tor-irc.dnsbl.oftc.net) Quit ()
[9:15] * yanzheng (~zhyan@182.139.22.205) Quit (Ping timeout: 480 seconds)
[9:17] * pvh_sa (~pvh@137.158.201.206) has joined #ceph
[9:17] * analbeard (~shw@support.memset.com) has joined #ceph
[9:18] * bj0rnar (~Bjornar@109.247.131.38) has joined #ceph
[9:20] * Vale (~Doodlepie@torsrvu.snydernet.net) has joined #ceph
[9:20] * b0e (~aledermue@213.95.25.82) has joined #ceph
[9:24] * allaok (~allaok@machine107.orange-labs.com) has joined #ceph
[9:25] * pvh_sa (~pvh@137.158.201.206) Quit (Ping timeout: 480 seconds)
[9:26] * owlbot (~supybot@pct-empresas-50.uc3m.es) Quit (Remote host closed the connection)
[9:28] * shtnik (~shtnik@188.120.252.178) Quit (Ping timeout: 480 seconds)
[9:28] * treenerd_ (~gsulzberg@cpe90-146-148-47.liwest.at) Quit (Quit: treenerd_)
[9:29] * dgurtner (~dgurtner@178.197.231.16) has joined #ceph
[9:29] * wjw-freebsd (~wjw@smtp.digiware.nl) has joined #ceph
[9:31] * pvh_sa (~pvh@137.158.201.206) has joined #ceph
[9:32] * fsimonce (~simon@host85-7-dynamic.49-82-r.retail.telecomitalia.it) has joined #ceph
[9:33] * RayTracer (~RayTracer@host-81-190-123-128.gdynia.mm.pl) has joined #ceph
[9:34] * erwan_taf (~erwan@46.231.131.178) has joined #ceph
[9:34] * lx0 (~aoliva@lxo.user.oftc.net) has joined #ceph
[9:35] * erwan_taf (~erwan@46.231.131.178) Quit ()
[9:35] * hyperbaba (~hyperbaba@private.neobee.net) has joined #ceph
[9:35] * erwan_taf (~erwan@46.231.131.178) has joined #ceph
[9:39] * pvh_sa (~pvh@137.158.201.206) Quit (Ping timeout: 480 seconds)
[9:40] * pvh_sa (~pvh@137.158.201.206) has joined #ceph
[9:41] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[9:43] * TMM (~hp@178-84-46-106.dynamic.upc.nl) Quit (Quit: Ex-Chat)
[9:48] * rdias (~rdias@2001:8a0:749a:d01:243c:1b7f:ce1a:5fe5) has joined #ceph
[9:48] * pvh_sa (~pvh@137.158.201.206) Quit (Ping timeout: 480 seconds)
[9:49] * drankis (~drankis__@89.111.13.198) has joined #ceph
[9:49] * drankis (~drankis__@89.111.13.198) Quit ()
[9:50] * Vale (~Doodlepie@76GAACPAL.tor-irc.dnsbl.oftc.net) Quit ()
[9:50] * dusti (~Sigma@93.115.95.205) has joined #ceph
[9:50] * adun153 (~ljtirazon@112.198.101.249) Quit (Read error: Connection reset by peer)
[9:50] * adun153 (~ljtirazon@112.198.101.249) has joined #ceph
[9:53] * jcsp (~jspray@82-71-16-249.dsl.in-addr.zen.co.uk) Quit (Ping timeout: 480 seconds)
[9:54] * MACscr|lappy (~MACscrlap@c-73-9-230-5.hsd1.il.comcast.net) has joined #ceph
[9:57] * allaok (~allaok@machine107.orange-labs.com) has left #ceph
[9:58] * garphy`aw is now known as garphy
[9:59] * Miouge_ (~Miouge@94.136.92.20) has joined #ceph
[10:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Remote host closed the connection)
[10:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[10:02] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[10:02] * pvh_sa (~pvh@137.158.201.206) has joined #ceph
[10:03] * Miouge (~Miouge@94.136.92.20) Quit (Ping timeout: 480 seconds)
[10:03] * Miouge_ is now known as Miouge
[10:05] * erwan_taf (~erwan@46.231.131.178) Quit (Ping timeout: 480 seconds)
[10:05] * RayTracer (~RayTracer@host-81-190-123-128.gdynia.mm.pl) Quit (Remote host closed the connection)
[10:09] * DV__ (~veillard@2001:41d0:a:f29f::1) has joined #ceph
[10:11] * pvh_sa (~pvh@137.158.201.206) Quit (Ping timeout: 480 seconds)
[10:11] * LeaChim (~LeaChim@host86-171-90-242.range86-171.btcentralplus.com) has joined #ceph
[10:13] * jordanP (~jordan@204.13-14-84.ripe.coltfrance.com) has joined #ceph
[10:15] * efirs (~firs@c-50-185-70-125.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[10:16] * DV (~veillard@2001:41d0:a:f29f::1) Quit (Ping timeout: 480 seconds)
[10:17] * nardial (~ls@dslb-178-011-177-180.178.011.pools.vodafone-ip.de) Quit (Quit: Leaving)
[10:19] * pvh_sa (~pvh@6c-88-14-d6-a1-4c.eduroam.uct.ac.za) has joined #ceph
[10:20] * dusti (~Sigma@4MJAACVGF.tor-irc.dnsbl.oftc.net) Quit ()
[10:20] * erwan_taf (~erwan@46.231.131.179) has joined #ceph
[10:20] * danielsj (~Swompie`@hessel0.torservers.net) has joined #ceph
[10:21] * allaok1 (~allaok@161.105.181.113) has joined #ceph
[10:21] * allaok1 (~allaok@161.105.181.113) has left #ceph
[10:22] * allaok2 (~allaok@machine107.orange-labs.com) has joined #ceph
[10:23] * TMM (~hp@185.5.122.2) has joined #ceph
[10:25] * shyu (~Shanzhi@119.254.120.66) Quit (Remote host closed the connection)
[10:25] * zhaochao (~zhaochao@125.39.112.15) Quit (Ping timeout: 480 seconds)
[10:26] * shyu (~Shanzhi@119.254.120.66) has joined #ceph
[10:26] * zhaochao (~zhaochao@124.202.191.135) has joined #ceph
[10:27] * zhaochao (~zhaochao@124.202.191.135) Quit ()
[10:29] * shyu (~Shanzhi@119.254.120.66) Quit (Remote host closed the connection)
[10:30] * zhaochao (~zhaochao@124.202.191.135) has joined #ceph
[10:30] * shyu (~Shanzhi@119.254.120.66) has joined #ceph
[10:33] * zhaochao (~zhaochao@124.202.191.135) Quit ()
[10:33] * ins0mnia (~textual@81-224-115-76-no238.tbcn.telia.com) Quit (Ping timeout: 480 seconds)
[10:35] * zhaochao (~zhaochao@124.202.191.135) has joined #ceph
[10:36] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Remote host closed the connection)
[10:37] * smokedmeets (~smokedmee@c-73-158-201-226.hsd1.ca.comcast.net) Quit (Quit: smokedmeets)
[10:37] * masteroman (~ivan@93-142-238-7.adsl.net.t-com.hr) has joined #ceph
[10:37] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[10:39] * yanzheng1 (~zhyan@182.139.20.43) Quit (Quit: This computer has gone to sleep)
[10:47] * ade (~abradshaw@guest.dyn.rm.dk) has joined #ceph
[10:50] * danielsj (~Swompie`@7V7AACU62.tor-irc.dnsbl.oftc.net) Quit ()
[10:50] * OODavo (~rushworld@relay1.cavefelem.com) has joined #ceph
[10:53] * smokedmeets (~smokedmee@c-73-158-201-226.hsd1.ca.comcast.net) has joined #ceph
[10:54] * adun153 (~ljtirazon@112.198.101.249) Quit (Read error: Connection reset by peer)
[10:55] * jcsp (~jspray@82-71-16-249.dsl.in-addr.zen.co.uk) has joined #ceph
[10:57] * wjw-freebsd (~wjw@smtp.digiware.nl) Quit (Ping timeout: 480 seconds)
[11:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Remote host closed the connection)
[11:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[11:08] * pvh_sa (~pvh@6c-88-14-d6-a1-4c.eduroam.uct.ac.za) Quit (Ping timeout: 480 seconds)
[11:16] * overclk (~vshankar@121.244.87.117) Quit (Quit: Zzzzzzzz...)
[11:20] * OODavo (~rushworld@84ZAACUM0.tor-irc.dnsbl.oftc.net) Quit ()
[11:20] * TomyLobo (~Inverness@76GAACPDL.tor-irc.dnsbl.oftc.net) has joined #ceph
[11:27] * desmond1984 (~desmond19@88.207.202.141) has joined #ceph
[11:28] * swami1 (~swami@49.38.0.28) Quit (Quit: Leaving.)
[11:28] * allaok2 (~allaok@machine107.orange-labs.com) Quit (Remote host closed the connection)
[11:32] * erwan_taf (~erwan@46.231.131.179) Quit (Ping timeout: 480 seconds)
[11:34] * swami1 (~swami@49.38.0.28) has joined #ceph
[11:34] * owlbot (~supybot@pct-empresas-50.uc3m.es) has joined #ceph
[11:36] * Miouge (~Miouge@94.136.92.20) Quit (Quit: Miouge)
[11:38] * allaok (~allaok@machine107.orange-labs.com) has joined #ceph
[11:38] * owlbot (~supybot@pct-empresas-50.uc3m.es) Quit (Remote host closed the connection)
[11:39] * Miouge (~Miouge@94.136.92.20) has joined #ceph
[11:41] * shylesh (~shylesh@121.244.87.118) Quit (Ping timeout: 480 seconds)
[11:43] * owlbot (~supybot@pct-empresas-50.uc3m.es) has joined #ceph
[11:44] * Miouge (~Miouge@94.136.92.20) Quit (Quit: Miouge)
[11:47] * Concubidated (~Adium@c-50-173-245-118.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[11:47] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Remote host closed the connection)
[11:47] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[11:50] * TomyLobo (~Inverness@76GAACPDL.tor-irc.dnsbl.oftc.net) Quit ()
[11:50] * verbalins (~djidis__@93.115.95.206) has joined #ceph
[11:53] * RayTracer (~RayTracer@153.19.7.39) has joined #ceph
[11:54] * shylesh (~shylesh@121.244.87.124) has joined #ceph
[11:55] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Ping timeout: 480 seconds)
[11:56] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit (Ping timeout: 480 seconds)
[11:57] * owlbot` (~supybot@pct-empresas-50.uc3m.es) has joined #ceph
[11:57] * allaok (~allaok@machine107.orange-labs.com) has left #ceph
[12:01] * adun153 (~ljtirazon@112.198.90.150) has joined #ceph
[12:01] * owlbot (~supybot@pct-empresas-50.uc3m.es) Quit (Remote host closed the connection)
[12:01] * owlbot` (~supybot@pct-empresas-50.uc3m.es) Quit (Remote host closed the connection)
[12:02] * erwan_taf (~erwan@46.231.131.178) has joined #ceph
[12:04] * pvh_sa (~pvh@6c-88-14-d6-a1-4c.eduroam.uct.ac.za) has joined #ceph
[12:05] * pabluk__ is now known as pabluk_
[12:08] * owlbot (~supybot@pct-empresas-50.uc3m.es) has joined #ceph
[12:08] * MACscr|lappy (~MACscrlap@c-73-9-230-5.hsd1.il.comcast.net) Quit (Ping timeout: 480 seconds)
[12:12] * MACscr|lappy (~MACscrlap@c-73-9-230-5.hsd1.il.comcast.net) has joined #ceph
[12:12] * pvh_sa (~pvh@6c-88-14-d6-a1-4c.eduroam.uct.ac.za) Quit (Ping timeout: 480 seconds)
[12:14] * pvh_sa (~pvh@6c-88-14-d6-a1-4c.eduroam.uct.ac.za) has joined #ceph
[12:16] * derjohn_mobi (~aj@2001:6f8:1337:0:2964:1974:9f34:487b) has joined #ceph
[12:20] * verbalins (~djidis__@7V7AACU8J.tor-irc.dnsbl.oftc.net) Quit ()
[12:22] * pvh_sa (~pvh@6c-88-14-d6-a1-4c.eduroam.uct.ac.za) Quit (Ping timeout: 480 seconds)
[12:24] * bildramer1 (~jacoo@orion.enn.lu) has joined #ceph
[12:25] * overclk (~vshankar@117.202.103.103) has joined #ceph
[12:26] * adun153 (~ljtirazon@112.198.90.150) Quit (Ping timeout: 480 seconds)
[12:26] * adun153 (~ljtirazon@112.198.90.150) has joined #ceph
[12:28] * adun153 (~ljtirazon@112.198.90.150) Quit (Remote host closed the connection)
[12:31] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[12:31] * T1w (~jens@node3.survey-it.dk) Quit (Remote host closed the connection)
[12:31] * pvh_sa (~pvh@6c-88-14-d6-a1-4c.eduroam.uct.ac.za) has joined #ceph
[12:31] * zhaochao (~zhaochao@124.202.191.135) Quit (Quit: ChatZilla 0.9.92 [Iceweasel 44.0.2/20160214092551])
[12:32] * karnan (~karnan@121.244.87.117) Quit (Remote host closed the connection)
[12:33] * edgarsb (~edgarsb@195.13.191.117) has joined #ceph
[12:34] <edgarsb> Hello. It seems I cannot find description for rbd_op_threads parameter. Can anybody explain it?
[12:37] * Miouge (~Miouge@94.136.92.20) has joined #ceph
[12:38] * owlbot (~supybot@pct-empresas-50.uc3m.es) Quit (Remote host closed the connection)
[12:39] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Ping timeout: 480 seconds)
[12:39] * pvh_sa (~pvh@6c-88-14-d6-a1-4c.eduroam.uct.ac.za) Quit (Ping timeout: 480 seconds)
[12:40] * rendar (~I@host18-177-dynamic.20-87-r.retail.telecomitalia.it) has joined #ceph
[12:44] * owlbot (~supybot@pct-empresas-50.uc3m.es) has joined #ceph
[12:48] * pvh_sa (~pvh@6c-88-14-d6-a1-4c.eduroam.uct.ac.za) has joined #ceph
[12:54] * bildramer1 (~jacoo@76GAACPE7.tor-irc.dnsbl.oftc.net) Quit ()
[12:54] * ZombieTree (~ggg@hessel0.torservers.net) has joined #ceph
[12:55] * i_m (~ivan.miro@deibp9eh1--blueice4n2.emea.ibm.com) has joined #ceph
[12:55] * owlbot (~supybot@pct-empresas-50.uc3m.es) Quit (Remote host closed the connection)
[12:56] * pvh_sa (~pvh@6c-88-14-d6-a1-4c.eduroam.uct.ac.za) Quit (Ping timeout: 480 seconds)
[12:57] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[12:58] * MACscr|lappy (~MACscrlap@c-73-9-230-5.hsd1.il.comcast.net) Quit (Ping timeout: 480 seconds)
[13:00] * sankarshan (~sankarsha@122.166.146.91) has joined #ceph
[13:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Remote host closed the connection)
[13:02] * pvh_sa (~pvh@6c-88-14-d6-a1-4c.eduroam.uct.ac.za) has joined #ceph
[13:04] * owlbot (~supybot@pct-empresas-50.uc3m.es) has joined #ceph
[13:07] * sep (~sep@95.62-50-191.enivest.net) has joined #ceph
[13:10] * pvh_sa (~pvh@6c-88-14-d6-a1-4c.eduroam.uct.ac.za) Quit (Ping timeout: 480 seconds)
[13:11] * owlbot (~supybot@pct-empresas-50.uc3m.es) Quit (Remote host closed the connection)
[13:11] * shylesh (~shylesh@121.244.87.124) Quit (Ping timeout: 480 seconds)
[13:12] * pvh_sa (~pvh@6c-88-14-d6-a1-4c.eduroam.uct.ac.za) has joined #ceph
[13:12] * owlbot (~supybot@pct-empresas-50.uc3m.es) has joined #ceph
[13:14] * rdias (~rdias@2001:8a0:749a:d01:243c:1b7f:ce1a:5fe5) Quit (Ping timeout: 480 seconds)
[13:20] * pvh_sa (~pvh@6c-88-14-d6-a1-4c.eduroam.uct.ac.za) Quit (Ping timeout: 480 seconds)
[13:20] * shylesh (~shylesh@121.244.87.118) has joined #ceph
[13:24] * ZombieTree (~ggg@7V7AACU9J.tor-irc.dnsbl.oftc.net) Quit ()
[13:25] * madkiss (~madkiss@2001:6f8:12c3:f00f:1d78:ae15:320f:422e) Quit (Quit: Leaving.)
[13:26] * jrocha (~jrocha@p4FE4FA6F.dip0.t-ipconnect.de) has joined #ceph
[13:29] * Chrissi_ (~LorenXo@65-183-154-104-dhcp.burlingtontelecom.net) has joined #ceph
[13:30] * pvh_sa (~pvh@6c-88-14-d6-a1-4c.eduroam.uct.ac.za) has joined #ceph
[13:31] * jrocha (~jrocha@p4FE4FA6F.dip0.t-ipconnect.de) Quit (Quit: Leaving)
[13:33] * kefu is now known as kefu|afk
[13:38] * pvh_sa (~pvh@6c-88-14-d6-a1-4c.eduroam.uct.ac.za) Quit (Ping timeout: 480 seconds)
[13:42] * drankis (~drankis__@89.111.13.198) has joined #ceph
[13:42] * drankis (~drankis__@89.111.13.198) Quit (Remote host closed the connection)
[13:43] * sleinen (~Adium@2001:620:0:82::101) has joined #ceph
[13:45] * pam (~pam@193.106.183.1) has joined #ceph
[13:47] * pvh_sa (~pvh@6c-88-14-d6-a1-4c.eduroam.uct.ac.za) has joined #ceph
[13:48] * kanagaraj (~kanagaraj@121.244.87.117) Quit (Ping timeout: 480 seconds)
[13:50] * Kupo1 (~tyler.wil@23.111.254.159) has joined #ceph
[13:52] * valeech (~valeech@pool-108-44-162-111.clppva.fios.verizon.net) has joined #ceph
[13:54] * Kupo (~tyler.wil@23.111.254.159) Quit (Ping timeout: 480 seconds)
[13:54] * Chrissi_ (~LorenXo@84ZAACUQ7.tor-irc.dnsbl.oftc.net) Quit ()
[13:55] * pvh_sa (~pvh@6c-88-14-d6-a1-4c.eduroam.uct.ac.za) Quit (Ping timeout: 480 seconds)
[13:55] * anadrom (~PierreW@anonymous6.sec.nl) has joined #ceph
[13:56] * pvh_sa (~pvh@6c-88-14-d6-a1-4c.eduroam.uct.ac.za) has joined #ceph
[13:56] * dugravot6 (~dugravot6@dn-infra-04.lionnois.site.univ-lorraine.fr) has joined #ceph
[13:56] * kanagaraj (~kanagaraj@121.244.87.124) has joined #ceph
[13:58] <pam> Hi, I have question about networking in ceph.
[13:59] <pam> Let???s say I have OSD server which are connected with 1 x 40 Gbps interface for the public network and with 1 x 40 Gbps for the cluster network on different switches
[14:01] <pam> what happend when the let???s say when the swtich of the cluster network dies
[14:02] <pam> will the traffic of the cluster network swtich to the public link or will the osd???s be dead
[14:04] * pvh_sa (~pvh@6c-88-14-d6-a1-4c.eduroam.uct.ac.za) Quit (Ping timeout: 480 seconds)
[14:05] * georgem (~Adium@24.114.74.62) has joined #ceph
[14:06] * wyang (~wyang@114.111.166.47) has joined #ceph
[14:07] * georgem (~Adium@24.114.74.62) Quit ()
[14:07] * kanagaraj (~kanagaraj@121.244.87.124) Quit (Ping timeout: 480 seconds)
[14:07] * georgem (~Adium@206.108.127.16) has joined #ceph
[14:08] <skoude> Hi,, any idea how to recover from 2016-02-24 15:09:04.398726 mon.0 [INF] pgmap v508581: 20772 pgs: 7388 stale+active+remapped, 678 active+degraded, 3537 active+remapped, 9 active+degraded+remapped, 9160 stale+active+degraded; 70797 MB data, 84821 MB used, 96368 GB / 96451 GB avail; 8242/19752 objects degraded (41.727%)
[14:10] <skoude> or should I just wait for it to recover?
[14:11] * linuxkidd (~linuxkidd@170.sub-70-193-125.myvzw.com) has joined #ceph
[14:13] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[14:17] * shyu_ (~shyu@123.116.175.103) has joined #ceph
[14:20] * wyang (~wyang@114.111.166.47) Quit (Quit: This computer has gone to sleep)
[14:21] <skoude> the problem is just that it won't go anywhere..
[14:21] <skoude> the percentage has been sitting there for an hour
[14:24] * shylesh (~shylesh@121.244.87.118) Quit (Remote host closed the connection)
[14:25] * anadrom (~PierreW@7V7AACVAV.tor-irc.dnsbl.oftc.net) Quit ()
[14:28] * kawa2014 (~kawa@89.184.114.246) has joined #ceph
[14:28] * rdas (~rdas@121.244.87.116) Quit (Quit: Leaving)
[14:29] * Racpatel (~Racpatel@2601:87:3:3601::2d54) Quit (Quit: Leaving)
[14:33] * neurodrone_ (~neurodron@pool-100-35-67-57.nwrknj.fios.verizon.net) has joined #ceph
[14:35] <skoude> hmm.. it seemed like my crushmap was somehow faulty.. I assigned it back again..
[14:37] * wyang (~wyang@114.111.166.47) has joined #ceph
[14:37] * sankarshan (~sankarsha@122.166.146.91) Quit (Quit: Are you sure you want to quit this channel (Cancel/Ok) ?)
[14:38] * jclm (~jclm@static-100-38-70-130.nycmny.fios.verizon.net) has joined #ceph
[14:39] * jclm (~jclm@static-100-38-70-130.nycmny.fios.verizon.net) Quit ()
[14:40] * georgem (~Adium@206.108.127.16) Quit (Ping timeout: 480 seconds)
[14:43] * wyang (~wyang@114.111.166.47) Quit (Quit: This computer has gone to sleep)
[14:43] <Gugge-47527> skoude: "stale" is not good
[14:46] <skoude> yeap, I noticed :D
[14:46] * wjw-freebsd (~wjw@vpn.ecoracks.nl) has joined #ceph
[14:47] * EinstCrazy (~EinstCraz@101.85.63.39) has joined #ceph
[14:47] * wyang (~wyang@116.216.30.6) has joined #ceph
[14:55] * homosaur (~Deiz@185.100.86.86) has joined #ceph
[15:00] <zenpac> Does anyone have a best practices rule for when to add capacity? at some percentage limit? Absolute size?
[15:00] * wyang (~wyang@116.216.30.6) Quit (Quit: This computer has gone to sleep)
[15:00] <zenpac> I hear that disks get over-taxed at about 90%
[15:00] <skoude> I was just testing new cluster.. I try to reproduce it again...
[15:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Remote host closed the connection)
[15:01] * mhack (~mhack@66-168-117-78.dhcp.oxfr.ma.charter.com) has joined #ceph
[15:01] * shylesh (~shylesh@121.244.87.124) has joined #ceph
[15:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[15:02] * avib (~Ceph@al.secure.elitehosts.com) Quit (Quit: Leaving)
[15:03] * bara (~bara@213.175.37.12) has joined #ceph
[15:05] * georgem (~Adium@206.108.127.16) has joined #ceph
[15:07] * wyang (~wyang@116.216.30.6) has joined #ceph
[15:09] * homosaur (~Deiz@7V7AACVCO.tor-irc.dnsbl.oftc.net) Quit (Remote host closed the connection)
[15:11] * neurodrone_ (~neurodron@pool-100-35-67-57.nwrknj.fios.verizon.net) Quit (Quit: neurodrone_)
[15:11] * swami1 (~swami@49.38.0.28) Quit (Read error: Connection reset by peer)
[15:13] <IcePic> zenpac: for some storage stuff (not ceph at the time), we used to have a rule of thumb that went "start planning for expansion at 50% full, order hw at 75%"
[15:13] * swami1 (~swami@49.38.0.28) has joined #ceph
[15:15] * pam (~pam@193.106.183.1) Quit (Quit: pam)
[15:15] * wyang (~wyang@116.216.30.6) Quit (Quit: This computer has gone to sleep)
[15:17] <IcePic> running disks to 90% (regardless) is a bad habit. Most filesystems that actively work against fragmentation themselves stop doing that at 95% full, and it leaves very little time for setting up and getting new storage into place.
[15:18] <IcePic> on a higher level, lots of the "move stuff by making a snap and copying that over to destination and then loop over the remaining data until we can remove the source" will want to have some space for the snapshot, which can make it hard or impossible to move out data even when you have new space to move stuff to.
[15:18] <IcePic> really frustrating to have added space and not being able to ask the %&#%# system to move old data to it. (I'm looking at you, 3PAR!)
[15:28] * Racpatel (~Racpatel@2601:87:3:3601::2d54) has joined #ceph
[15:31] <zenpac> IcePic: I agree with that. I suppose I can have different levels of warnings.
[15:32] * bniver (~bniver@nat-pool-bos-u.redhat.com) has joined #ceph
[15:32] * shaunm (~shaunm@74.83.215.100) has joined #ceph
[15:33] <IcePic> It would very obviously depend on how much is 10% in your setup, how fast you can get hw in place and so on.
[15:33] <IcePic> its nice to make calculations based on last month and project how many days left the free space means in the current speed of usage increase so you can tell when you have 60 days left of storage, for instance.
[15:35] * EinstCrazy (~EinstCraz@101.85.63.39) Quit (Remote host closed the connection)
[15:36] * pam (~pam@193.106.183.1) has joined #ceph
[15:39] * swami1 (~swami@49.38.0.28) Quit (Quit: Leaving.)
[15:40] * naoto (~naotok@27.131.11.254) Quit (Quit: Leaving...)
[15:42] * alfredodeza (~alfredode@198.206.133.89) has left #ceph
[15:43] * harold (~hamiller@71-94-227-123.dhcp.mdfd.or.charter.com) has joined #ceph
[15:44] * harold (~hamiller@71-94-227-123.dhcp.mdfd.or.charter.com) Quit ()
[15:44] * LobsterRoll (~LobsterRo@140.247.242.44) has joined #ceph
[15:45] * LobsterRoll (~LobsterRo@140.247.242.44) has left #ceph
[15:49] * pvh_sa (~pvh@169-0-182-212.ip.afrihost.co.za) has joined #ceph
[15:51] * treenerd_ (~gsulzberg@cpe90-146-148-47.liwest.at) has joined #ceph
[15:57] <Miouge> Is there any down side to reducing the ???mon osd down out interval??? value? I was thinking somewhere between 5 to 30s
[15:59] * vikhyat (~vumrao@121.244.87.116) Quit (Quit: Leaving)
[16:00] * treenerd_ (~gsulzberg@cpe90-146-148-47.liwest.at) Quit (Ping timeout: 480 seconds)
[16:00] * linuxkidd (~linuxkidd@170.sub-70-193-125.myvzw.com) Quit (Ping timeout: 480 seconds)
[16:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Remote host closed the connection)
[16:01] * xarses (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[16:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[16:02] * dillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) Quit (Remote host closed the connection)
[16:02] <IcePic> a quick reboot that takes more than 5-30s will have the rest of the osds start "fixing" the missing data, and then some 45s after when your OSD came back up, they will reshuffle again, I suppose.
[16:06] <Miouge> So somewhere around reboot time + margin would be suitable?
[16:07] * bara (~bara@213.175.37.12) Quit (Ping timeout: 480 seconds)
[16:09] * kefu|afk is now known as kefu
[16:10] * Kayla (~brannmar@2.tor.exit.babylon.network) has joined #ceph
[16:10] * Pies (~Pies@srv229.opcja.pl) Quit (Ping timeout: 480 seconds)
[16:10] * pam (~pam@193.106.183.1) Quit (Quit: pam)
[16:11] * pam (~pam@193.106.183.1) has joined #ceph
[16:13] * linuxkidd (~linuxkidd@59.sub-70-193-100.myvzw.com) has joined #ceph
[16:13] <IcePic> well, its hard to cover all possible events, but the time before the cluster starts acting should probably cover the time you think a normal planned "outage" takes, just so it doesnt start churning for something like a planned reboot for updates.
[16:13] * brad_mssw (~brad@66.129.88.50) has joined #ceph
[16:14] * hyperbaba (~hyperbaba@private.neobee.net) Quit (Ping timeout: 480 seconds)
[16:19] * xarses (~xarses@64.124.158.100) has joined #ceph
[16:23] * georgem (~Adium@206.108.127.16) Quit (Ping timeout: 480 seconds)
[16:27] * TomasCZ (~TomasCZ@yes.tenlab.net) has joined #ceph
[16:28] * jotik (~7f000001@160.143.35.213.sta.estpak.ee) has joined #ceph
[16:34] * ibravo (~ibravo@72.83.69.64) has joined #ceph
[16:34] * pam (~pam@193.106.183.1) Quit (Quit: pam)
[16:37] * GeoTracer (~Geoffrey@41.77.153.99) Quit (Read error: Connection reset by peer)
[16:37] * GeoTracer (~Geoffrey@41.77.153.99) has joined #ceph
[16:39] * Kayla (~brannmar@7V7AACVE6.tor-irc.dnsbl.oftc.net) Quit ()
[16:39] * darkid (~Quackie@7V7AACVGF.tor-irc.dnsbl.oftc.net) has joined #ceph
[16:43] * gregmark (~Adium@68.87.42.115) has joined #ceph
[16:43] * bara (~bara@nat-pool-brq-t.redhat.com) has joined #ceph
[16:43] * pam (~pam@193.106.183.1) has joined #ceph
[16:43] * bara (~bara@nat-pool-brq-t.redhat.com) Quit (Remote host closed the connection)
[16:45] * ade (~abradshaw@guest.dyn.rm.dk) Quit (Ping timeout: 480 seconds)
[16:45] * bara (~bara@nat-pool-brq-t.redhat.com) has joined #ceph
[16:49] * enax (~enax@hq.ezit.hu) Quit (Ping timeout: 480 seconds)
[16:52] * haplo37 (~haplo37@199.91.185.156) has joined #ceph
[16:52] * jdillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) has joined #ceph
[16:58] * ibravo2 (~ibravo@72.198.142.104) has joined #ceph
[16:58] * zaitcev (~zaitcev@c-50-130-189-82.hsd1.nm.comcast.net) has joined #ceph
[16:59] * bvi (~bastiaan@185.56.32.1) has joined #ceph
[17:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Remote host closed the connection)
[17:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[17:03] * ibravo (~ibravo@72.83.69.64) Quit (Ping timeout: 480 seconds)
[17:04] * b0e (~aledermue@213.95.25.82) Quit (Quit: Leaving.)
[17:05] * analbeard (~shw@support.memset.com) Quit (Quit: Leaving.)
[17:09] * shtnik (~shtnik@37.49.162.23) has joined #ceph
[17:09] * MentalRay (~MentalRay@office-mtl1-nat-146-218-70-69.gtcomm.net) has joined #ceph
[17:09] * darkid (~Quackie@7V7AACVGF.tor-irc.dnsbl.oftc.net) Quit ()
[17:09] * Pommesgabel (~Corti^car@193.90.12.89) has joined #ceph
[17:13] * mattbenjamin (~mbenjamin@sccc-66-78-236-243.smartcity.com) has joined #ceph
[17:16] * MACscr|lappy (~MACscrlap@c-73-9-230-5.hsd1.il.comcast.net) has joined #ceph
[17:18] * kefu is now known as kefu|afk
[17:21] <boolman> anyone having issues with slow read/write in kvm vm? works fine on the physical host. http://pastebin.com/ySJFS8aY
[17:22] * bvi (~bastiaan@185.56.32.1) Quit (Quit: Ex-Chat)
[17:22] <boolman> 4.2.0-30-generic, libvirt 1.2.2, ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
[17:26] * shylesh (~shylesh@121.244.87.124) Quit (Remote host closed the connection)
[17:27] <edgarsb> generaly poor read performance
[17:27] <boolman> any tip on how to solve it? :)
[17:27] <edgarsb> 3 nodes, 6 2TB HDD + 1 SSD journal; 3x 128 GB cache tier
[17:27] <edgarsb> not yet
[17:28] <edgarsb> I got around 30 MB/sec in each VM
[17:28] <edgarsb> now removed cache tier and playing with primary-affinity
[17:28] <edgarsb> will have some results tomorrow
[17:28] <boolman> i get around 30 write and 50 read
[17:29] * wjw-freebsd (~wjw@vpn.ecoracks.nl) Quit (Ping timeout: 480 seconds)
[17:29] <edgarsb> writes were no issue for me, strange thou that physical hosts report low busy times for OSD disks
[17:30] <edgarsb> anyway, have to go now, I hope to get back here tomorrow
[17:30] * edgarsb away
[17:31] * kanagaraj (~kanagaraj@27.7.11.25) has joined #ceph
[17:31] * bjornar_ (~bjornar@ti0099a430-0908.bb.online.no) has joined #ceph
[17:32] * bjornar__ (~bjornar@ti0099a430-0908.bb.online.no) has joined #ceph
[17:34] * mattbenjamin (~mbenjamin@sccc-66-78-236-243.smartcity.com) Quit (Ping timeout: 480 seconds)
[17:34] * pam (~pam@193.106.183.1) Quit (Quit: pam)
[17:34] * bene_in_mtg (~bene@nat-pool-bos-t.redhat.com) has joined #ceph
[17:35] * fcape (~fcape@rrcs-97-77-228-30.sw.biz.rr.com) has joined #ceph
[17:36] * fcape (~fcape@rrcs-97-77-228-30.sw.biz.rr.com) Quit ()
[17:37] * kanagaraj (~kanagaraj@27.7.11.25) Quit (Quit: Leaving)
[17:37] * fcape (~fcape@rrcs-97-77-228-30.sw.biz.rr.com) has joined #ceph
[17:39] * Pommesgabel (~Corti^car@4MJAACVYR.tor-irc.dnsbl.oftc.net) Quit ()
[17:39] * dusti (~slowriot@62.210.105.116) has joined #ceph
[17:43] * vata (~vata@207.96.182.162) has joined #ceph
[17:43] * sudocat (~dibarra@192.185.1.20) has joined #ceph
[17:44] * krypto (~krypto@G68-121-13-209.sbcis.sbc.com) has joined #ceph
[17:51] * kawa2014 (~kawa@89.184.114.246) Quit (Ping timeout: 480 seconds)
[17:51] * linjan_ (~linjan@86.62.112.22) Quit (Ping timeout: 480 seconds)
[17:53] * eth00 (~r00t@70-130-76-23.lightspeed.rlghnc.sbcglobal.net) Quit (Ping timeout: 480 seconds)
[17:53] * georgem (~Adium@206.108.127.16) has joined #ceph
[17:54] * linuxkidd_ (~linuxkidd@221.sub-70-193-73.myvzw.com) has joined #ceph
[17:55] * sleinen1 (~Adium@130.59.94.198) has joined #ceph
[17:55] * linuxkidd (~linuxkidd@59.sub-70-193-100.myvzw.com) Quit (Ping timeout: 480 seconds)
[17:56] * pabluk_ is now known as pabluk__
[17:56] * sleinen2 (~Adium@2001:620:0:82::103) has joined #ceph
[17:58] * ircolle (~Adium@2601:285:201:2bf9:b8f5:27d0:d505:9564) has joined #ceph
[17:58] <MrFusion> this is what i get in my windows 7 kvm: http://puu.sh/niWjH/3ecdfd90b5.png
[17:59] * shtnik (~shtnik@37.49.162.23) Quit (Ping timeout: 480 seconds)
[18:00] * kawa2014 (~kawa@89.184.114.246) has joined #ceph
[18:00] * kanagaraj (~kanagaraj@27.7.11.25) has joined #ceph
[18:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Remote host closed the connection)
[18:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[18:02] * sleinen (~Adium@2001:620:0:82::101) Quit (Ping timeout: 480 seconds)
[18:03] * derjohn_mobi (~aj@2001:6f8:1337:0:2964:1974:9f34:487b) Quit (Ping timeout: 480 seconds)
[18:03] * sleinen1 (~Adium@130.59.94.198) Quit (Ping timeout: 480 seconds)
[18:04] * bara (~bara@nat-pool-brq-t.redhat.com) Quit (Quit: Bye guys! (??????????????????? ?????????)
[18:05] * MACscr|lappy (~MACscrlap@c-73-9-230-5.hsd1.il.comcast.net) Quit (Ping timeout: 480 seconds)
[18:06] * TMM (~hp@185.5.122.2) Quit (Quit: Ex-Chat)
[18:08] * shohn (~shohn@220.Red-79-146-26.dynamicIP.rima-tde.net) has joined #ceph
[18:08] * shohn (~shohn@220.Red-79-146-26.dynamicIP.rima-tde.net) has left #ceph
[18:09] * dusti (~slowriot@4MJAACVZ6.tor-irc.dnsbl.oftc.net) Quit ()
[18:09] * RayTracer (~RayTracer@153.19.7.39) Quit (Remote host closed the connection)
[18:09] * osuka_ (~n0x1d@tor-exit.gansta93.com) has joined #ceph
[18:19] * sleinen2 (~Adium@2001:620:0:82::103) Quit (Ping timeout: 480 seconds)
[18:20] * linuxkidd_ (~linuxkidd@221.sub-70-193-73.myvzw.com) Quit (Ping timeout: 480 seconds)
[18:21] * dugravot6 (~dugravot6@dn-infra-04.lionnois.site.univ-lorraine.fr) Quit (Quit: Leaving.)
[18:21] * dvanders (~dvanders@46.227.20.178) has joined #ceph
[18:24] * kanagaraj (~kanagaraj@27.7.11.25) Quit (Quit: Leaving)
[18:25] * vasu (~vasu@c-73-231-60-138.hsd1.ca.comcast.net) has joined #ceph
[18:27] * kefu|afk (~kefu@114.92.107.250) Quit (Max SendQ exceeded)
[18:27] * treenerd_ (~gsulzberg@cpe90-146-148-47.liwest.at) has joined #ceph
[18:27] * kefu (~kefu@114.92.107.250) has joined #ceph
[18:29] * swami1 (~swami@27.7.172.147) has joined #ceph
[18:29] * linuxkidd_ (~linuxkidd@205.sub-70-193-124.myvzw.com) has joined #ceph
[18:29] * jotik (~7f000001@160.143.35.213.sta.estpak.ee) has left #ceph
[18:39] * osuka_ (~n0x1d@4MJAACV1K.tor-irc.dnsbl.oftc.net) Quit ()
[18:39] * Pettis (~Knuckx@phoolandevi.tor-exit.calyxinstitute.org) has joined #ceph
[18:45] * derjohn_mobi (~aj@x590dfb93.dyn.telefonica.de) has joined #ceph
[18:48] * treenerd_ (~gsulzberg@cpe90-146-148-47.liwest.at) Quit (Ping timeout: 480 seconds)
[18:48] * TMM (~hp@178-84-46-106.dynamic.upc.nl) has joined #ceph
[18:49] * swami1 (~swami@27.7.172.147) Quit (Quit: Leaving.)
[18:50] * kefu (~kefu@114.92.107.250) Quit (Read error: Connection reset by peer)
[18:51] * kefu (~kefu@114.92.107.250) has joined #ceph
[18:53] * mykola (~Mikolaj@91.225.200.81) has joined #ceph
[18:55] * kanagaraj (~kanagaraj@27.7.11.25) has joined #ceph
[18:55] * ircolle (~Adium@2601:285:201:2bf9:b8f5:27d0:d505:9564) Quit (Quit: Leaving.)
[18:56] * jordanP (~jordan@204.13-14-84.ripe.coltfrance.com) Quit (Remote host closed the connection)
[18:56] * kefu (~kefu@114.92.107.250) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[18:58] * dillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) has joined #ceph
[18:59] * sleinen (~Adium@2001:620:1000:3:a65e:60ff:fedb:f305) has joined #ceph
[19:00] * kawa2014 (~kawa@89.184.114.246) Quit (Quit: Leaving)
[19:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Remote host closed the connection)
[19:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[19:04] * johnavp1989 (~jpetrini@69.160.43.38) has joined #ceph
[19:08] * linuxkidd (~linuxkidd@173.sub-70-193-123.myvzw.com) has joined #ceph
[19:09] * Pettis (~Knuckx@7V7AACVKD.tor-irc.dnsbl.oftc.net) Quit ()
[19:12] * karnan (~karnan@106.51.19.29) has joined #ceph
[19:13] * linuxkidd_ (~linuxkidd@205.sub-70-193-124.myvzw.com) Quit (Ping timeout: 480 seconds)
[19:13] * bene_in_mtg is now known as bene2
[19:14] * bret1 (~mr_flea@198.134.125.78) has joined #ceph
[19:14] * branto (~branto@nat-pool-brq-t.redhat.com) Quit (Quit: Leaving.)
[19:15] * neerbeer (~neerbeer@2600:1003:b10e:f976:5876:2616:f9e8:f732) has joined #ceph
[19:19] * diq (~diq@2620:11c:f:2:c23f:d5ff:fe62:112c) has joined #ceph
[19:19] * linuxkidd (~linuxkidd@173.sub-70-193-123.myvzw.com) Quit (Ping timeout: 480 seconds)
[19:20] * desmond1984 (~desmond19@88.207.202.141) Quit (Remote host closed the connection)
[19:22] * linuxkidd (~linuxkidd@176.sub-70-193-122.myvzw.com) has joined #ceph
[19:24] * kanagaraj (~kanagaraj@27.7.11.25) Quit (Quit: Leaving)
[19:27] * georgem (~Adium@206.108.127.16) Quit (Ping timeout: 480 seconds)
[19:27] * Be-El (~blinke@nat-router.computational.bio.uni-giessen.de) Quit (Quit: Leaving.)
[19:29] * thansen (~thansen@17.253.sfcn.org) has joined #ceph
[19:31] * BrianA (~BrianA@fw-rw.shutterfly.com) has joined #ceph
[19:31] * BrianA (~BrianA@fw-rw.shutterfly.com) Quit (Read error: Connection reset by peer)
[19:34] * bniver (~bniver@nat-pool-bos-u.redhat.com) Quit (Remote host closed the connection)
[19:35] * BrianA (~BrianA@fw-rw.shutterfly.com) has joined #ceph
[19:38] * karnan (~karnan@106.51.19.29) Quit (Read error: Connection timed out)
[19:40] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[19:40] * lx0 (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[19:41] * i_m (~ivan.miro@deibp9eh1--blueice4n2.emea.ibm.com) Quit (Ping timeout: 480 seconds)
[19:41] * RayTracer (~RayTracer@host-81-190-123-128.gdynia.mm.pl) has joined #ceph
[19:42] * dgurtner (~dgurtner@178.197.231.16) Quit (Ping timeout: 480 seconds)
[19:44] * bret1 (~mr_flea@198.134.125.78) Quit ()
[19:44] * hoopy (~hoopy@tor2r.ins.tor.net.eu.org) has joined #ceph
[19:49] * ffilz (~ffilz@c-76-115-190-27.hsd1.or.comcast.net) has joined #ceph
[19:49] * krypto (~krypto@G68-121-13-209.sbcis.sbc.com) Quit (Remote host closed the connection)
[19:52] * RayTracer (~RayTracer@host-81-190-123-128.gdynia.mm.pl) Quit (Remote host closed the connection)
[19:53] * georgem (~Adium@206.108.127.16) has joined #ceph
[19:55] * jdillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) Quit (Quit: jdillaman)
[19:56] * bene_in_mtg (~bene@nat-pool-bos-t.redhat.com) has joined #ceph
[19:56] * RayTracer (~RayTracer@host-81-190-123-128.gdynia.mm.pl) has joined #ceph
[19:57] * Concubidated (~Adium@c-50-173-245-118.hsd1.ca.comcast.net) has joined #ceph
[19:58] * The_Ball (~pi@80.203.114.75) Quit (Read error: Connection reset by peer)
[19:59] * LobsterRoll (~LobsterRo@140.247.242.44) has joined #ceph
[19:59] * bene2 (~bene@nat-pool-bos-t.redhat.com) Quit (Ping timeout: 480 seconds)
[20:00] * secate_ (~Secate@196-210-31-224.dynamic.isadsl.co.za) has joined #ceph
[20:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Remote host closed the connection)
[20:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[20:01] * Geph (~Geoffrey@169-1-168-32.ip.afrihost.co.za) has joined #ceph
[20:04] * RayTracer (~RayTracer@host-81-190-123-128.gdynia.mm.pl) Quit (Ping timeout: 480 seconds)
[20:05] * linuxkidd (~linuxkidd@176.sub-70-193-122.myvzw.com) Quit (Ping timeout: 480 seconds)
[20:06] * linjan_ (~linjan@176.195.241.98) has joined #ceph
[20:07] * secate (~Secate@196-210-31-52.dynamic.isadsl.co.za) Quit (Ping timeout: 480 seconds)
[20:08] <johnavp1989> Hi guys I'm following this guide to restore a journal http://ceph.com/planet/ceph-recover-osds-after-ssd-journal-failure/
[20:08] <johnavp1989> But I don't have the original UUID
[20:08] <johnavp1989> How can I restore the journal without it?
[20:13] * mykola (~Mikolaj@91.225.200.81) Quit (Ping timeout: 480 seconds)
[20:14] * hoopy (~hoopy@4MJAACV42.tor-irc.dnsbl.oftc.net) Quit ()
[20:14] * xENO_2 (~ricin@4MJAACV58.tor-irc.dnsbl.oftc.net) has joined #ceph
[20:14] * overclk (~vshankar@117.202.103.103) Quit (Quit: Zzzzzzzz...)
[20:15] * linuxkidd (~linuxkidd@202.sub-97-34-70.myvzw.com) has joined #ceph
[20:19] * mykola (~Mikolaj@91.225.200.81) has joined #ceph
[20:20] <flaf> johnavp1989: personally I don't have this file journal_uuid in my osds too.
[20:21] * RayTracer (~RayTracer@host-81-190-123-128.gdynia.mm.pl) has joined #ceph
[20:22] <flaf> Maybe I'm wrong but I seems to me that journal_uuid is just the UUID of the journal partition.
[20:22] * bniver (~bniver@nat-pool-bos-u.redhat.com) has joined #ceph
[20:23] <flaf> *it seems
[20:23] * dgurtner (~dgurtner@217-162-119-191.dynamic.hispeed.ch) has joined #ceph
[20:23] <flaf> in this case you can see it with the ??? blkid ??? command for instance (in a linux).
[20:28] <neurodrone> Has anyone played with qemu+rbd setup?
[20:29] * pam (~pam@host246-56-dynamic.48-82-r.retail.telecomitalia.it) has joined #ceph
[20:32] * simeon_ (~simeon@apu0.dhcp.meraka.csir.co.za) has joined #ceph
[20:32] * wjw-freebsd (~wjw@smtp.digiware.nl) has joined #ceph
[20:34] * daviddcc (~dcasier@84.197.151.77.rev.sfr.net) has joined #ceph
[20:34] <simeon_> hi, is there a way to convince a librbd client (qemu) to bind to a specific interface (to originate TCP connections from that source IP)?
[20:34] * masterom1 (~ivan@93-142-208-91.adsl.net.t-com.hr) has joined #ceph
[20:36] * ade (~abradshaw@80.62.20.66) has joined #ceph
[20:38] * masteroman (~ivan@93-142-238-7.adsl.net.t-com.hr) Quit (Ping timeout: 480 seconds)
[20:39] * masteroman (~ivan@93-142-238-207.adsl.net.t-com.hr) has joined #ceph
[20:39] * neerbeer (~neerbeer@2600:1003:b10e:f976:5876:2616:f9e8:f732) Quit (Ping timeout: 480 seconds)
[20:42] * linuxkidd (~linuxkidd@202.sub-97-34-70.myvzw.com) Quit (Ping timeout: 480 seconds)
[20:43] * masterom1 (~ivan@93-142-208-91.adsl.net.t-com.hr) Quit (Ping timeout: 480 seconds)
[20:44] * xENO_2 (~ricin@4MJAACV58.tor-irc.dnsbl.oftc.net) Quit ()
[20:44] * djidis__ (~sardonyx@nooduitgang.schmutzig.org) has joined #ceph
[20:45] * fcape_ (~fcape@38.67.19.2) has joined #ceph
[20:47] * fcape (~fcape@rrcs-97-77-228-30.sw.biz.rr.com) Quit (Ping timeout: 480 seconds)
[20:48] * mattbenjamin (~mbenjamin@sccc-66-78-236-243.smartcity.com) has joined #ceph
[20:48] * diq (~diq@2620:11c:f:2:c23f:d5ff:fe62:112c) has left #ceph
[20:49] * ade (~abradshaw@80.62.20.66) Quit (Ping timeout: 480 seconds)
[20:51] * linuxkidd (~linuxkidd@52.sub-70-193-68.myvzw.com) has joined #ceph
[20:53] <neurodrone> simeon_: Have you checked the public_network parameter?
[20:53] * pam (~pam@host246-56-dynamic.48-82-r.retail.telecomitalia.it) Quit (Read error: Connection reset by peer)
[20:54] * pam (~pam@nat1.unibz.it) has joined #ceph
[20:54] * valeech (~valeech@pool-108-44-162-111.clppva.fios.verizon.net) Quit (Quit: valeech)
[20:55] <simeon_> neurodrone: I have a routed network
[20:56] * fcape (~fcape@38.67.19.2) has joined #ceph
[20:56] <simeon_> neurodrone: does public_network in ceph.conf on a client influence source address selection?
[20:57] <neurodrone> It should aid the client in interface selection.
[20:57] <neurodrone> Should be worthy to try out.
[20:58] * fcape_ (~fcape@38.67.19.2) Quit (Ping timeout: 480 seconds)
[21:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Remote host closed the connection)
[21:01] * pam (~pam@nat1.unibz.it) Quit (Read error: Connection reset by peer)
[21:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[21:02] * fcape_ (~fcape@38.67.19.2) has joined #ceph
[21:04] * fcape (~fcape@38.67.19.2) Quit (Ping timeout: 480 seconds)
[21:04] * valeech (~valeech@pool-108-44-162-111.clppva.fios.verizon.net) has joined #ceph
[21:08] * dvanders_ (~dvanders@46.227.20.178) has joined #ceph
[21:09] * mattbenjamin (~mbenjamin@sccc-66-78-236-243.smartcity.com) Quit (Ping timeout: 480 seconds)
[21:10] * DV__ (~veillard@2001:41d0:a:f29f::1) Quit (Remote host closed the connection)
[21:11] * DV (~veillard@2001:41d0:a:f29f::1) has joined #ceph
[21:14] * djidis__ (~sardonyx@4MJAACV7G.tor-irc.dnsbl.oftc.net) Quit ()
[21:15] * dvanders (~dvanders@46.227.20.178) Quit (Ping timeout: 480 seconds)
[21:15] * bene_in_mtg (~bene@nat-pool-bos-t.redhat.com) Quit (Quit: Konversation terminated!)
[21:15] * fcape (~fcape@38.67.19.2) has joined #ceph
[21:17] * fcape_ (~fcape@38.67.19.2) Quit (Ping timeout: 480 seconds)
[21:21] <simeon_> neurodrone: I'll give that a go, thanks for the pointer
[21:22] <neurodrone> Np.
[21:22] * bniver (~bniver@nat-pool-bos-u.redhat.com) Quit (Remote host closed the connection)
[21:27] <ibravo2> question: how can I create a client and admin token from command line?
[21:27] <ibravo2> I need to include it into a Openstack configuration file to be deployed using TripleO
[21:29] <neurodrone> Is the ceph documentation unclear about it?
[21:30] <ibravo2> not clear to me. Installed 9.2 but the documentation talks about ceph-deploy that is no longer there
[21:31] * scuttlemonkey is now known as scuttle|afk
[21:31] <ibravo2> http://docs.ceph.com/docs/master/install/
[21:32] <ibravo2> These are all the packages I'm looking at (Centos 7) http://download.ceph.com/rpm-infernalis/el7/x86_64/
[21:35] * LobsterRoll_ (~LobsterRo@65.112.8.204) has joined #ceph
[21:36] * RayTracer (~RayTracer@host-81-190-123-128.gdynia.mm.pl) Quit (Quit: Leaving...)
[21:38] * LobsterRoll (~LobsterRo@140.247.242.44) Quit (Ping timeout: 480 seconds)
[21:38] * LobsterRoll_ is now known as LobsterRoll
[21:39] <ibravo2> Found something here: http://docs.ceph.com/docs/master/man/8/ceph-authtool/#cmdoption-ceph-authtool--gen-key
[21:43] * masteroman (~ivan@93-142-238-207.adsl.net.t-com.hr) Quit (Quit: WeeChat 1.4)
[21:43] * georgem1 (~Adium@206.108.127.16) has joined #ceph
[21:43] * srk (~oftc-webi@32.97.110.56) has joined #ceph
[21:44] * georgem (~Adium@206.108.127.16) Quit (Read error: Connection reset by peer)
[21:44] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[21:45] <srk> hi, anyone used ceph-disk to add osds to the cluster? I'm trying to deploy a cluster based on bcache devices.
[21:45] * mykola (~Mikolaj@91.225.200.81) Quit (Quit: away)
[21:51] * bene_in_mtg (~bene@nat-pool-bos-t.redhat.com) has joined #ceph
[21:52] * fcape_ (~fcape@rrcs-97-77-228-30.sw.biz.rr.com) has joined #ceph
[21:54] * fcape (~fcape@38.67.19.2) Quit (Ping timeout: 480 seconds)
[21:54] * dgurtner (~dgurtner@217-162-119-191.dynamic.hispeed.ch) Quit (Ping timeout: 480 seconds)
[21:55] * markednmbr1 (~Diego@cpc1-lewi13-2-0-cust267.2-4.cable.virginm.net) has joined #ceph
[21:55] <markednmbr1> hello
[21:55] <markednmbr1> I'm considering replacing my sata drives with ssd's to speed up ceph (I'm getting about 15MB/s at the moment)
[21:56] <markednmbr1> anyone got any views on differen brands? of course im thinking consumer ssd's rather than expensive ones....
[21:56] * richard (~oftc-webi@50-233-46-101-static.hfc.comcastbusiness.net) has joined #ceph
[21:56] <Kvisle> markednmbr1: have you confirmed that the disks are the bottleneck in your setup?
[21:57] <markednmbr1> hmm no, just assumed, what else could be?
[21:58] <markednmbr1> nothing else can go that slow can it? :)
[21:58] <markednmbr1> could it be bad config?
[21:58] * richard (~oftc-webi@50-233-46-101-static.hfc.comcastbusiness.net) Quit ()
[21:59] * fdmanana (~fdmanana@2001:8a0:6dfd:6d01:4963:611d:d4e9:c457) Quit (Ping timeout: 480 seconds)
[22:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Remote host closed the connection)
[22:01] <Kvisle> 15MB/s, is that write, read, or both?
[22:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[22:02] <markednmbr1> write, only tested write
[22:04] <lincolnb> w/win 2
[22:04] <lincolnb> ugh, can't IRC.
[22:05] <Kvisle> markednmbr1: how many osd servers do you have, and how many osds in each? and the pool you've benchmarked; how many replicas does it have?
[22:05] <markednmbr1> 4 servers, 3 osds in each
[22:05] <markednmbr1> n+1
[22:05] <Kvisle> how are they connected to the network?
[22:06] <markednmbr1> gigabit
[22:06] * fcape_ (~fcape@rrcs-97-77-228-30.sw.biz.rr.com) Quit (Ping timeout: 480 seconds)
[22:06] <Kvisle> do you use any kind of bonding? any kind of cluster network?
[22:06] <markednmbr1> 2 nics in each active/backup to difference switches
[22:08] <markednmbr1> well its actually 4 nics in each but 2 in an active/backup bond for ceph
[22:08] <Kvisle> two unused?
[22:08] <markednmbr1> 2 in active/backup for "lan"
[22:09] <Kvisle> so the switches in question, are these two top-of-rack switches that go back to a core switch or something like that?
[22:11] <Kvisle> keep in mind that active/backup-bonding may favor a different switch on the different servers ... thus making traffic between servers on the same switch have to pass an uplink
[22:11] <Kvisle> and keep in mind that for every write to an osd, it will make replica count outbound connections and perform the replications there
[22:11] <lincolnb> markednmbr1: are you talking MB/s in rados bench or via one of the Ceph interfaces (RBD, CephFS, etc..)?
[22:12] <Kvisle> it's not that difficult to get network saturation if one does not keep these things in mind.
[22:12] <markednmbr1> I have set them all to be primary on the same switch but you could be right I should do more network checks
[22:13] <Kvisle> are you using cluster networking in ceph?
[22:13] <Kvisle> so that the backfills/replication goes through a different channel than the client traffic?
[22:13] <markednmbr1> lincolnb, it's rbd - testing is actually in a ganeti VM
[22:13] <markednmbr1> Kvisle, I am using a cluster network but im sending client traffic over the same network
[22:14] <Kvisle> I'd do some measuring, and just pull out a calculator and see if I could find the bottleneck
[22:14] <Kvisle> remember to measure the counters on the uplink ports if possible
[22:14] <markednmbr1> cool I'll have another look at the network side before I throw money at it
[22:15] <markednmbr1> cheers for the pointer. gotta jet for now!
[22:15] <markednmbr1> cya
[22:15] <Kvisle> striping writes across 4 sata-drives should give you more than 15MB/s :P
[22:15] <srk> Kvisle: Do you know whether any other command need to be run for osds to be part of the cluster, after doing a ceph-disk activate (using hammer 0.94.6)
[22:15] * markednmbr1 (~Diego@cpc1-lewi13-2-0-cust267.2-4.cable.virginm.net) Quit (Quit: Leaving)
[22:16] <Kvisle> srk: never used ceph-disk, I just create the keyring and add them to the crush map and start the osd
[22:16] <Kvisle> like this: http://redpill-linpro.com/sysadvent/2015/12/18/stateless-osd-servers.html
[22:17] <srk> start osd as-in , service ceph-osd start?
[22:18] <ibravo2> markednmbr1: check this out for SSD tests http://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/
[22:18] * pvh_sa (~pvh@169-0-182-212.ip.afrihost.co.za) Quit (Ping timeout: 480 seconds)
[22:23] * pvh_sa (~pvh@169-0-182-212.ip.afrihost.co.za) has joined #ceph
[22:23] * n0x1d (~Defaultti@us1x.mullvad.net) has joined #ceph
[22:24] * dgurtner (~dgurtner@217-162-119-191.dynamic.hispeed.ch) has joined #ceph
[22:33] <aarontc> Does anyone know where the list of mds tunables/runtime parameters lives? I can't find it in the docs
[22:41] * scuttle|afk is now known as scuttlemonkey
[22:52] * Rickus (~Rickus@office.protected.ca) Quit (Read error: Connection reset by peer)
[22:53] <lurbs> aarontc: http://docs.ceph.com/docs/master/cephfs/mds-config-ref/
[22:53] * n0x1d (~Defaultti@76GAACPYK.tor-irc.dnsbl.oftc.net) Quit ()
[22:53] * JWilbur (~Spessu@192.42.115.101) has joined #ceph
[22:53] <aarontc> thanks, lurbs. Not sure why I didn't think to look under cephfs :)
[22:54] <lurbs> Also, 'ceph --show-config' will print out everything.
[22:54] <lurbs> Although without the descriptions.
[22:54] <aarontc> hmm I actually didn't know about that option. cool :)
[22:57] * LobsterRoll (~LobsterRo@65.112.8.204) Quit (Quit: LobsterRoll)
[23:00] * Rickus (~Rickus@office.protected.ca) has joined #ceph
[23:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Remote host closed the connection)
[23:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[23:02] * MentalRay (~MentalRay@office-mtl1-nat-146-218-70-69.gtcomm.net) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[23:06] * DV (~veillard@2001:41d0:a:f29f::1) Quit (Remote host closed the connection)
[23:07] * DV (~veillard@2001:41d0:a:f29f::1) has joined #ceph
[23:12] * mhack is now known as mhack|afk
[23:17] * fdmanana (~fdmanana@2001:8a0:6dfd:6d01:4963:611d:d4e9:c457) has joined #ceph
[23:17] * pvh_sa (~pvh@169-0-182-212.ip.afrihost.co.za) Quit (Ping timeout: 480 seconds)
[23:18] * shtnik (~shtnik@37.49.162.23) has joined #ceph
[23:19] * johnavp1989 (~jpetrini@69.160.43.38) Quit (Read error: Connection reset by peer)
[23:19] * brad_mssw (~brad@66.129.88.50) Quit (Quit: Leaving)
[23:22] * georgem1 (~Adium@206.108.127.16) Quit (Quit: Leaving.)
[23:23] * JWilbur (~Spessu@84ZAACVBB.tor-irc.dnsbl.oftc.net) Quit ()
[23:23] * Kurimus1 (~starcoder@torsrvq.snydernet.net) has joined #ceph
[23:24] * mattbenjamin (~mbenjamin@sccc-66-78-236-243.smartcity.com) has joined #ceph
[23:26] * gregmark (~Adium@68.87.42.115) Quit (Quit: Leaving.)
[23:29] * MentalRay (~MentalRay@MTRLPQ42-1176054809.sdsl.bell.ca) has joined #ceph
[23:29] * fdmanana (~fdmanana@2001:8a0:6dfd:6d01:4963:611d:d4e9:c457) Quit (Ping timeout: 480 seconds)
[23:29] * haplo37 (~haplo37@199.91.185.156) Quit (Remote host closed the connection)
[23:31] * Geph (~Geoffrey@169-1-168-32.ip.afrihost.co.za) Quit (Read error: Connection reset by peer)
[23:36] * gregmark (~Adium@68.87.42.115) has joined #ceph
[23:39] * vata (~vata@207.96.182.162) Quit (Quit: Leaving.)
[23:39] * shtnik (~shtnik@37.49.162.23) Quit (Ping timeout: 480 seconds)
[23:40] * ronrib (~boswortr@45.32.242.135) Quit (Quit: Leaving.)
[23:43] * fcape (~fcape@107-220-57-73.lightspeed.austtx.sbcglobal.net) has joined #ceph
[23:47] * srk (~oftc-webi@32.97.110.56) Quit (Ping timeout: 480 seconds)
[23:48] * Geph (~Geoffrey@169-1-168-32.ip.afrihost.co.za) has joined #ceph
[23:51] * dillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) Quit (Ping timeout: 480 seconds)
[23:53] * Kurimus1 (~starcoder@7V7AACVRY.tor-irc.dnsbl.oftc.net) Quit ()
[23:53] * rapedex (~maku@46.166.188.220) has joined #ceph
[23:54] * Racpatel (~Racpatel@2601:87:3:3601::2d54) Quit (Ping timeout: 480 seconds)

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.