#ceph IRC Log

Index

IRC Log for 2015-12-06

Timestamps are in GMT/BST.

[0:11] * alexxy (~alexxy@2001:470:1f14:106::2) Quit (Ping timeout: 480 seconds)
[0:11] * yanzheng (~zhyan@171.216.95.21) Quit (Quit: This computer has gone to sleep)
[0:14] * EinstCrazy (~EinstCraz@117.13.201.130) has joined #ceph
[0:15] * yanzheng (~zhyan@171.216.95.21) has joined #ceph
[0:16] * onlyanegg (~tcouto@c-73-162-126-221.hsd1.ca.comcast.net) has joined #ceph
[0:18] * MACscr (~Adium@2601:247:4101:a0be:ecd3:8e5a:b8df:f6fd) has joined #ceph
[0:20] * alexxy (~alexxy@2001:470:1f14:106::2) has joined #ceph
[0:21] * tsg (~tgohad@192.55.54.40) Quit (Remote host closed the connection)
[0:22] * EinstCrazy (~EinstCraz@117.13.201.130) Quit (Ping timeout: 480 seconds)
[0:27] * yanzheng (~zhyan@171.216.95.21) Quit (Quit: This computer has gone to sleep)
[0:40] * yanzheng (~zhyan@171.216.95.21) has joined #ceph
[0:46] * yanzheng (~zhyan@171.216.95.21) Quit (Quit: This computer has gone to sleep)
[0:50] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[0:50] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[0:56] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[0:56] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[0:59] * garphy is now known as garphy`aw
[1:00] * alexxy (~alexxy@2001:470:1f14:106::2) Quit (Ping timeout: 480 seconds)
[1:08] * alexxy (~alexxy@2001:470:1f14:106::2) has joined #ceph
[1:16] * osso (~osso@sgp01-1-78-233-150-179.fbx.proxad.net) has joined #ceph
[1:19] * stiopa (~stiopa@cpc73828-dals21-2-0-cust630.20-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[1:21] * diegows (~diegows@190.190.21.75) has joined #ceph
[1:28] * alexxy (~alexxy@2001:470:1f14:106::2) Quit (Ping timeout: 480 seconds)
[1:36] * alexxy (~alexxy@2001:470:1f14:106::2) has joined #ceph
[1:39] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[1:39] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[1:42] * diegows (~diegows@190.190.21.75) Quit (Ping timeout: 480 seconds)
[1:47] * Concubidated (~Adium@pool-98-119-93-148.lsanca.fios.verizon.net) has joined #ceph
[1:58] * onlyanegg (~tcouto@c-73-162-126-221.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[2:04] * tsg (~tgohad@jfdmzpr03-ext.jf.intel.com) has joined #ceph
[2:14] * rendar (~I@host184-182-dynamic.26-79-r.retail.telecomitalia.it) Quit (Quit: std::lower_bound + std::less_equal *works* with a vector without duplicates!)
[2:22] * yanzheng (~zhyan@171.216.95.21) has joined #ceph
[2:25] * yanzheng (~zhyan@171.216.95.21) Quit ()
[2:40] * dyasny (~dyasny@104.158.24.36) has joined #ceph
[2:40] * osso (~osso@sgp01-1-78-233-150-179.fbx.proxad.net) Quit (Quit: osso)
[2:45] * kalmisto (~Kalado@37.48.120.135) has joined #ceph
[3:03] * brutuscat (~brutuscat@105.34.133.37.dynamic.jazztel.es) Quit (Ping timeout: 480 seconds)
[3:08] * C2J (~c2j@114.93.152.67) has joined #ceph
[3:09] * C2J_ (~c2j@114.93.152.67) has joined #ceph
[3:09] * C2J (~c2j@114.93.152.67) Quit (Read error: Connection reset by peer)
[3:13] * EinstCrazy (~EinstCraz@117.13.201.130) has joined #ceph
[3:15] * kalmisto (~Kalado@7V7AABQPC.tor-irc.dnsbl.oftc.net) Quit ()
[3:27] * brutuscat (~brutuscat@105.34.133.37.dynamic.jazztel.es) has joined #ceph
[3:32] * C2J_ (~c2j@114.93.152.67) Quit (Remote host closed the connection)
[3:32] * C2J (~c2j@114.93.152.67) has joined #ceph
[3:50] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[3:50] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[3:51] * Mika_c (~Mika@36-227-14-162.dynamic-ip.hinet.net) has joined #ceph
[3:56] * aj__ (~aj@x4db04a49.dyn.telefonica.de) has joined #ceph
[3:56] * Steki (~steki@cable-89-216-227-238.dynamic.sbb.rs) has joined #ceph
[4:03] * BManojlovic (~steki@cable-89-216-235-151.dynamic.sbb.rs) Quit (Ping timeout: 480 seconds)
[4:03] * derjohn_mobi (~aj@x4db1b06e.dyn.telefonica.de) Quit (Ping timeout: 480 seconds)
[4:04] * dyasny (~dyasny@104.158.24.36) Quit (Ping timeout: 480 seconds)
[4:15] * davidz1 (~davidz@2605:e000:1313:8003:20f1:6dfa:24cd:5f85) has joined #ceph
[4:21] * davidz (~davidz@2605:e000:1313:8003:d8a1:e9ce:4eab:5b68) Quit (Ping timeout: 480 seconds)
[4:35] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[4:35] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[4:36] * Kyso_ (~sixofour@80.82.64.233) has joined #ceph
[4:40] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[4:40] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[4:55] * georgem (~Adium@75-119-226-89.dsl.teksavvy.com) has joined #ceph
[5:03] * \ask (~ask@oz.develooper.com) has joined #ceph
[5:06] * Vacuum_ (~Vacuum@88.130.211.62) has joined #ceph
[5:06] * Kyso_ (~sixofour@6YRAABCH2.tor-irc.dnsbl.oftc.net) Quit ()
[5:09] * brutuscat (~brutuscat@105.34.133.37.dynamic.jazztel.es) Quit (Ping timeout: 480 seconds)
[5:12] * Vacuum__ (~Vacuum@i59F79CAA.versanet.de) Quit (Ping timeout: 480 seconds)
[5:34] * sileht (~sileht@sileht.net) Quit (Ping timeout: 480 seconds)
[5:38] * tsg_ (~tgohad@134.134.139.72) has joined #ceph
[5:38] * tsg (~tgohad@jfdmzpr03-ext.jf.intel.com) Quit (Remote host closed the connection)
[6:01] * brutuscat (~brutuscat@105.34.133.37.dynamic.jazztel.es) has joined #ceph
[6:09] * brutuscat (~brutuscat@105.34.133.37.dynamic.jazztel.es) Quit (Ping timeout: 480 seconds)
[6:09] * aleksag (~AotC@46.166.190.189) has joined #ceph
[6:13] * overclk (~vshankar@59.93.67.125) has joined #ceph
[6:13] * georgem (~Adium@75-119-226-89.dsl.teksavvy.com) Quit (Quit: Leaving.)
[6:29] * Mika_ (~Mika@36-227-14-162.dynamic-ip.hinet.net) has joined #ceph
[6:30] * Mika_ (~Mika@36-227-14-162.dynamic-ip.hinet.net) Quit ()
[6:30] * Mika_c (~Mika@36-227-14-162.dynamic-ip.hinet.net) Quit (Quit: Leaving)
[6:31] * Mika_c (~Mika@36-227-14-162.dynamic-ip.hinet.net) has joined #ceph
[6:39] * overclk (~vshankar@59.93.67.125) Quit (Ping timeout: 480 seconds)
[6:39] * aleksag (~AotC@4Z9AABSNY.tor-irc.dnsbl.oftc.net) Quit ()
[6:45] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[6:45] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[6:56] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[6:56] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[6:57] * Nicola-1980 (~Nicola-19@2-234-77-205.ip222.fastwebnet.it) Quit (Ping timeout: 480 seconds)
[6:59] * wjw-freebsd (~wjw@smtp.digiware.nl) has joined #ceph
[7:41] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[7:41] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[7:43] * hroussea (~hroussea@000200d7.user.oftc.net) has joined #ceph
[8:02] * brutuscat (~brutuscat@105.34.133.37.dynamic.jazztel.es) has joined #ceph
[8:10] * brutuscat (~brutuscat@105.34.133.37.dynamic.jazztel.es) Quit (Ping timeout: 480 seconds)
[8:15] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[8:23] * yanzheng (~zhyan@171.216.95.21) has joined #ceph
[8:30] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Remote host closed the connection)
[8:36] * linjan (~linjan@176.195.163.243) has joined #ceph
[8:42] * overclk (~vshankar@59.93.67.125) has joined #ceph
[8:55] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[8:55] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[8:59] * overclk (~vshankar@59.93.67.125) Quit (Quit: No windows for this server)
[9:00] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[9:00] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[9:05] * bjozet (~bjozet@82-183-17-144.customers.ownit.se) Quit (Quit: leaving)
[9:09] * rotbeard (~redbeard@aftr-95-222-29-74.unity-media.net) has joined #ceph
[9:29] * sileht (~sileht@sileht.net) has joined #ceph
[9:40] * stiopa (~stiopa@cpc73828-dals21-2-0-cust630.20-2.cable.virginm.net) has joined #ceph
[9:44] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[9:44] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[9:48] * Izanagi (~VampiricP@104.238.192.58) has joined #ceph
[9:50] * Nicola-1980 (~Nicola-19@2-234-77-205.ip222.fastwebnet.it) has joined #ceph
[9:54] * Mons (~manens@relay.manens.org) has joined #ceph
[9:55] * Mons (~manens@relay.manens.org) has left #ceph
[9:55] * Mons (~manens@relay.manens.org) has joined #ceph
[10:01] * tsg_ (~tgohad@134.134.139.72) Quit (Remote host closed the connection)
[10:01] * tsg_ (~tgohad@134.134.139.72) has joined #ceph
[10:02] * DV_ (~veillard@2001:41d0:1:d478::1) Quit (Ping timeout: 480 seconds)
[10:04] * brutuscat (~brutuscat@105.34.133.37.dynamic.jazztel.es) has joined #ceph
[10:09] * olid111118 (~olid1982@p54848EBE.dip0.t-ipconnect.de) has joined #ceph
[10:12] * brutuscat (~brutuscat@105.34.133.37.dynamic.jazztel.es) Quit (Ping timeout: 480 seconds)
[10:16] * Infected (infected@peon.lantrek.fi) has joined #ceph
[10:17] * tsg__ (~tgohad@192.55.54.40) has joined #ceph
[10:17] * tsg_ (~tgohad@134.134.139.72) Quit (Remote host closed the connection)
[10:18] * Izanagi (~VampiricP@104.238.192.58) Quit ()
[10:23] * garphy`aw is now known as garphy
[10:35] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[10:35] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[10:35] * TMM (~hp@178-84-46-106.dynamic.upc.nl) Quit (Remote host closed the connection)
[10:40] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[10:40] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[10:45] * garphy is now known as garphy`aw
[10:45] * rendar (~I@host112-177-dynamic.10-87-r.retail.telecomitalia.it) has joined #ceph
[11:03] * wjw-freebsd (~wjw@smtp.digiware.nl) Quit (Ping timeout: 480 seconds)
[11:19] * TMM (~hp@178-84-46-106.dynamic.upc.nl) has joined #ceph
[11:25] * tsg__ (~tgohad@192.55.54.40) Quit (Remote host closed the connection)
[11:25] * tsg__ (~tgohad@192.55.54.40) has joined #ceph
[11:26] * nardial (~ls@dslb-178-006-188-146.178.006.pools.vodafone-ip.de) has joined #ceph
[11:36] <cetex> hm
[11:39] <cetex> still getting these errors, all osd's seems to be stuck in an infinite loop.. http://pastebin.com/KCg5n1JM
[11:39] <cetex> now the network is 20Gbe per host
[11:41] <cetex> ceph-osd has setup and torn down ~500k connections over the last ~10-20minutes
[11:41] <cetex> on one host
[11:41] <cetex> same on the other one
[11:42] <cetex> ceph-osd is using ~300% cpu
[11:44] <cetex> all 4 of them is using ~300% cpu
[11:44] <cetex> each
[11:44] <cetex> ;>
[11:44] * Gecko1986 (~totalworm@195-154-231-147.rev.poneytelecom.eu) has joined #ceph
[11:44] <cetex> any ideas?
[11:45] <destrudo> networking problem?
[11:45] <cetex> not that i know of
[11:45] <destrudo> What's your topology
[11:45] <destrudo> bonded 10g to a switch?
[11:45] <cetex> yeah.
[11:45] <cetex> with l3/l4 hashing so connections sticks to one path for the duration of the session.
[11:46] <cetex> it's actually aristas with mlag.
[11:46] <cetex> server has one link per switch
[11:46] <destrudo> can you see what OSD's are utilizing a crapload of cpu?
[11:47] <cetex> i see that all of them are using a crapload of cpu
[11:47] <destrudo> are you dockering?
[11:47] <cetex> indeed. but with --net=host
[11:47] <destrudo> https://www.mail-archive.com/ceph-users@lists.ceph.com/msg14841.html
[11:48] <T1> oh, evil
[11:48] <cetex> yeah.. i don't think he was using --net=host
[11:49] <cetex> since the container has no network virtualization in my case
[11:49] <cetex> dumped 1second of traffic now, 40MB ...
[11:49] <cetex> <- wireshark
[11:49] <destrudo> yay
[11:49] <T1> and you are using the same hash on the switches as on the hosts?
[11:50] <destrudo> the ceph logs should be logging active connections
[11:50] <destrudo> maybe 'could' and not should
[11:50] <cetex> T1: actually, maybe not on the switches. but in that case they should do it mac-based.
[11:51] <cetex> but to exclude that i'll just kill one switch
[11:51] <T1> cetex: randomly dropping tcp connections and 802.3ad with mixes hashing is not good..
[11:51] <T1> mixed even
[11:51] <cetex> yeah. maybe not.
[11:51] <cetex> but i just killed switch 2
[11:52] <cetex> i'll restart monitors + osd's
[11:56] <cetex> wiped all nodes..
[11:59] <T1> what about host network driver and firmware?
[12:01] <cetex> ixgbe, intel x520
[12:01] <cetex> had the same problem on standard 1gbe nics earlier as well
[12:01] * Gecko1986 (~totalworm@5P6AABTWC.tor-irc.dnsbl.oftc.net) Quit (Read error: Connection reset by peer)
[12:01] <cetex> no idea about firmware, dell provided nics
[12:01] <T1> updated driver and firmware?
[12:01] <T1> ah, check anyway
[12:01] * johnhunter (~hunter@222.29.39.73) has joined #ceph
[12:01] <destrudo> lol
[12:01] <cetex> although, i don't understand why ceph would fail while all other stuff works ;)
[12:02] <destrudo> I'd focus on a configuration issue
[12:02] <cetex> if it's nic firmware and stuff.
[12:02] <destrudo> not hardware or modules
[12:02] <destrudo> lol
[12:02] <cetex> yeah.
[12:02] <T1> I've got "pure" intel version of X710 a few months ago - new firmware + driver for RHEL 7.1 was a really really good idea
[12:02] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[12:02] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[12:03] <T1> firmware was 5 or 6 versions old - changelog said they fixed a lot of things
[12:03] <cetex> hm
[12:03] <cetex> the problems seems to occur when i launch the second osd on one node
[12:03] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[12:03] <cetex> i wonder if it's related to running osd and monitor on the same nodes..
[12:03] <destrudo> should not be an issue
[12:03] <destrudo> but log the mon
[12:03] <cetex> yeah. shouldn't.
[12:03] <destrudo> there might be some loop
[12:04] <destrudo> debug styles logs
[12:04] <cetex> but i'll test runing the only on the other nodes for now
[12:04] <T1> and the stock i40e driver in thel 7.1 is 1.0.3 something, while the latest from intel is/was 1.3.39.1
[12:04] <T1> afk
[12:05] * badone (~badone@66.187.239.16) Quit (Remote host closed the connection)
[12:05] * brutuscat (~brutuscat@105.34.133.37.dynamic.jazztel.es) has joined #ceph
[12:06] <destrudo> It'd be odd if docker was causing confuse with sockets
[12:06] * olid111119 (~olid1982@p54848EBE.dip0.t-ipconnect.de) has joined #ceph
[12:06] <cetex> yeah. i'm running ubuntu kernel 4.0.9-040009-generic, ixgbe version 4.0.1
[12:06] * olid111118 (~olid1982@p54848EBE.dip0.t-ipconnect.de) Quit (Read error: Connection reset by peer)
[12:06] <destrudo> I don't know enough about it's deep deep love to figure it out
[12:07] <cetex> so.. wiped cluster again.
[12:07] <cetex> monitors one one end, osd's on the slaves.
[12:08] <destrudo> well, not to not figure it out, but to actually tell you something immediately
[12:09] <cetex> yeah. it does stuff usually, sets up a bridge, nat and stuff
[12:09] <cetex> but i've disabled all of dockers messy networking
[12:09] <destrudo> that I'm aware of
[12:09] <cetex> so it's running in the same network namespace as the host
[12:09] <destrudo> I'm talking about the deep stuff
[12:09] <cetex> :)
[12:09] <destrudo> Like, if you open a socket inside a docker instance
[12:09] <destrudo> what is it doing
[12:10] <cetex> since it's running in the default namespace it isn't any different than the host
[12:10] <destrudo> are you 100% sure of that?
[12:10] * badone (~badone@66.187.239.16) has joined #ceph
[12:10] <destrudo> I would assume it'd be the most sensible thing, yes
[12:10] <destrudo> but I don't know
[12:10] <destrudo> what's the status?
[12:11] <cetex> two osd's on one node up
[12:11] <cetex> works ok
[12:11] <cetex> as expected
[12:11] <cetex> launching one on a second node
[12:12] <cetex> the two i started first aren't logging anything special, seems ok
[12:12] <cetex> the third one i started bailed out
[12:12] <destrudo> bailed out?
[12:12] <destrudo> if this fails, you should pull any bonding/whatnot
[12:13] <destrudo> and do single cable to each node
[12:13] <destrudo> just to see
[12:13] <cetex> http://pastebin.com/M63PJVFE
[12:13] <destrudo> There might be some horrible interaction
[12:13] <cetex> this is without bonding
[12:13] <cetex> or, well, it's configured, but only one interface active
[12:13] * brutuscat (~brutuscat@105.34.133.37.dynamic.jazztel.es) Quit (Ping timeout: 480 seconds)
[12:13] <destrudo> hmm
[12:14] <cetex> so that's the third osd launched
[12:14] <cetex> that pastebin
[12:15] <cetex> this is one of the other two who seem alright
[12:15] <cetex> http://pastebin.com/atQS7a8n
[12:15] <cetex> the third osd is on another host
[12:17] * mps (~Popz@85.159.237.199) has joined #ceph
[12:18] <destrudo> this totally smells of a networking issue
[12:18] <cetex> so: first pastebin (http://pastebin.com/M63PJVFE) is from one osd on a host where two osd's are running, everything seems fine i then launched a third osd on another host, and that one bailed out (http://pastebin.com/atQS7a8n)
[12:18] <destrudo> can you enable debugging on the OSD's?
[12:18] <cetex> hm..
[12:18] <cetex> yeah.
[12:18] <destrudo> debug ms = 10
[12:18] <destrudo> in the OSD configs
[12:19] <cetex> cool. all of them?
[12:19] <cetex> or just the broken one?
[12:19] <cetex> i'll do all..
[12:19] <destrudo> all.
[12:19] <destrudo> what docker version are you rolling with?
[12:19] <cetex> newest, one sec
[12:19] <destrudo> you're seeing shitloads of traffic, rite?
[12:19] <cetex> 1.9.1
[12:20] <cetex> seems like ~100Mbit of connection attempts...
[12:20] <destrudo> what's the host OS?
[12:20] <cetex> a stripped down ubuntu
[12:20] <cetex> trusty
[12:20] <cetex> both on the host and in the container
[12:22] <cetex> moar logs coming
[12:23] * C2J_ (~c2j@114.93.152.67) has joined #ceph
[12:24] * Wielebny (~Icedove@cl-927.waw-01.pl.sixxs.net) has joined #ceph
[12:25] <cetex> damn amount of logs..
[12:25] <destrudo> are you using --privileged=true?
[12:26] <cetex> good qustion. will see
[12:26] <destrudo> in the docker
[12:26] <destrudo> if not, try it.
[12:26] <cetex> nope
[12:26] <cetex> ran debug ms = 10 for ~20seconds on the working one and ~10seconds on the broken one
[12:26] <cetex> ~500MB logs..
[12:26] <destrudo> yeah
[12:27] <destrudo> you'll need to find the good parts on your own
[12:28] <cetex> ;)
[12:28] <destrudo> I don't even think the privileged statement will do anything
[12:29] <destrudo> but it's worth trying (to me) since I have no idea how this stuff works at a low level
[12:29] <cetex> yeah. sure.
[12:29] <destrudo> looks like it's used to mount devfs stuff
[12:29] <destrudo> not configure networking
[12:29] <destrudo> but what the hell do I know
[12:29] <cetex> fun fun. 2347332 = loglines on broken
[12:30] * C2J__ (~c2j@114.93.152.67) has joined #ceph
[12:30] * C2J (~c2j@114.93.152.67) Quit (Ping timeout: 480 seconds)
[12:31] * rotbeard (~redbeard@aftr-95-222-29-74.unity-media.net) Quit (Ping timeout: 480 seconds)
[12:36] * C2J_ (~c2j@114.93.152.67) Quit (Ping timeout: 480 seconds)
[12:37] * garphy`aw is now known as garphy
[12:38] <destrudo> are you running more than one ceph OSD/mon instance per host?
[12:38] <destrudo> eh, nm
[12:38] <destrudo> I don't know if that's valid
[12:39] <cetex> two osd's per host
[12:40] <cetex> pastebin has javascript. it tries to parse my paste
[12:40] <cetex> (only 10k lines though)
[12:40] <cetex> but yeah. slowish
[12:40] * musca (musca@tyrael.eu) has joined #ceph
[12:41] <destrudo> try running one OSD per host
[12:41] * musca (musca@tyrael.eu) has left #ceph
[12:42] <cetex> one per host seems to be working
[12:43] * musca (musca@tyrael.eu) has joined #ceph
[12:43] <destrudo> hmmm
[12:43] * musca (musca@tyrael.eu) has left #ceph
[12:44] <destrudo> try one mon and one osd per host
[12:44] <cetex> so, when i start two osd's on one host the osd on the third host starts crapping
[12:44] <destrudo> hmm
[12:45] <cetex> from 58 tcp sessions to 100k in ~1.5seconds
[12:45] <destrudo> I don't think docker's network is very transparent
[12:45] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[12:45] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Remote host closed the connection)
[12:45] * osso (~osso@sgp01-1-78-233-150-179.fbx.proxad.net) has joined #ceph
[12:47] * davidz (~davidz@2605:e000:1313:8003:20f1:6dfa:24cd:5f85) has joined #ceph
[12:47] * davidz1 (~davidz@2605:e000:1313:8003:20f1:6dfa:24cd:5f85) Quit (Read error: Connection reset by peer)
[12:47] <cetex> yeah. there may be something weird going on, but i'd like to know why ceph bails out when everything else works ;)
[12:47] * mps (~Popz@85.159.237.199) Quit ()
[12:48] <cetex> stuff that will fail is if an osd needs to find the pid of another osd and such
[12:48] <cetex> but as long as it's just connecting to the mon, finding all other osd's that way and then connects to the other osd's by ip it should "just work" imho..
[12:50] <destrudo> so lets us continue
[12:50] <destrudo> Every OSD will be using the host's IP
[12:50] <destrudo> that is each docker instance using the same IP
[12:51] <destrudo> in your logs
[12:51] <destrudo> tcpdump
[12:51] <destrudo> Are the OSD's attempting to connect to only one port?
[12:53] <cetex> hm.
[12:53] <cetex> just did a test with privileged
[12:53] <cetex> now two osd's running on a single host crapped out 100%
[12:53] <destrudo> did it play nice?
[12:53] <destrudo> oh
[12:53] <cetex> ah.. only temporarily apparently
[12:55] <cetex> and now stuff seems to work actually
[12:55] <destrudo> yay
[12:56] <cetex> so.. what's ceph doing that requires privileged?
[12:56] <cetex> ;>
[12:56] <destrudo> maybe seeing if a port is available?
[12:56] <destrudo> check out your tcpdump
[12:56] <destrudo> see if there's only one port getting used on that IP
[12:57] <destrudo> I'm wondering if it has some issue determining whether or not a port is free
[12:57] <destrudo> and is attempting to open a socket on the already used port
[12:58] <cetex> it can see all listening ports on the hosts actually.
[12:58] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[12:58] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[12:59] <destrudo> well, then my theory is wack
[12:59] <cetex> aaaand no. it died again when i redeployed *
[12:59] <destrudo> aw.
[12:59] <destrudo> odd that it played nice for a little while tho
[13:00] <destrudo> I see By default, daemons bind to ports within the 6800:7300 range. You may configure this range at your discretion. Before configuring your IP tables, check the default iptables configuration.
[13:00] <destrudo> whoa
[13:00] <destrudo> I see 6802 and 6803 getting requests in the pipe push
[13:01] <cetex> http://pastebin.com/SU3ed7hG
[13:02] <cetex> grepped logs from one osd on one host to another osd on another host
[13:02] * C2J (~c2j@114.93.152.67) has joined #ceph
[13:02] <cetex> :0 is b0rked
[13:02] <cetex> why is the port 0?
[13:02] <destrudo> the guy in the first post has logs that show the same thing
[13:03] <cetex> yeah
[13:03] <destrudo> are you running any other docker containers?
[13:03] <destrudo> eh
[13:04] <cetex> i am.
[13:04] <cetex> same setup
[13:04] <cetex> mesos, aurora, some transcoding tasks
[13:04] <destrudo> on the same system?
[13:04] <cetex> yeah.
[13:05] <destrudo> nuke em'
[13:05] <cetex> sure. i'm pretty sure it doesn't matter ;)
[13:05] <cetex> but i'll do it.
[13:05] <destrudo> me too, but I have no fuckin' idea at this point
[13:05] <cetex> nuked
[13:05] <destrudo> short of staring at everything myself I don't think I've got anything
[13:06] <destrudo> and I have shit to do
[13:06] <destrudo> lol
[13:07] <cetex> ;D
[13:07] <cetex> still the same.
[13:08] <cetex> new container, infernalis
[13:08] <cetex> testing
[13:08] <cetex> soon. takes time.
[13:09] * C2J__ (~c2j@114.93.152.67) Quit (Ping timeout: 480 seconds)
[13:15] * C2J (~c2j@114.93.152.67) Quit (Remote host closed the connection)
[13:15] <cetex> missing uuidgen in the container, i guess infernalis has different dpkg dependencies. :)
[13:15] <cetex> rebuilding.
[13:15] * agsha (~sharath.g@103.5.134.169) has joined #ceph
[13:24] <cetex> wiped *; pushing infernalis monitors
[13:25] * brutuscat (~brutuscat@105.34.133.37.dynamic.jazztel.es) has joined #ceph
[13:26] <cetex> ceph v9.2.0, still same..
[13:28] * linjan (~linjan@176.195.163.243) Quit (Ping timeout: 480 seconds)
[13:30] <destrudo> I'm gonna just blame docker, but I have no solution.
[13:31] <destrudo> maybe create a real bridge and configure your container instances to use it?
[13:32] <cetex> real bridge?
[13:32] <cetex> i don't want a bridge at all ;)
[13:32] * DV_ (~veillard@2001:41d0:a:f29f::1) has joined #ceph
[13:32] <cetex> it messes with stuff to much.
[13:32] * nils_ (~nils_@doomstreet.collins.kg) has joined #ceph
[13:35] <cetex> it costs a bunch of cpu cycles and the goal is to actually push close to 20Gbit on these hosts.
[13:35] <destrudo> these obviously aren't production systems, why not try
[13:35] <destrudo> sure
[13:35] <cetex> and i don't understand how a bridge would help since it adds a layer :)
[13:36] <destrudo> as I said, don't know how docker plays, adding that layer might make it play nice
[13:36] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[13:36] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[13:37] * linjan (~linjan@176.195.163.243) has joined #ceph
[13:39] <cetex> yeah. docker does a lot of stuff, but it's just using standard functionality in the linux kernel underneath.
[13:41] <cetex> saw that the osd complains about lsb-release
[13:41] <cetex> adding that to the container
[13:41] <cetex> i don't understand why it wants to know the release
[13:41] <cetex> though..
[13:42] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[13:42] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[13:45] * osso (~osso@sgp01-1-78-233-150-179.fbx.proxad.net) Quit (Quit: osso)
[13:45] * Wielebny (~Icedove@cl-927.waw-01.pl.sixxs.net) Quit (Remote host closed the connection)
[13:48] <cetex> nope, still same...
[13:48] <cetex> bah
[13:50] <cetex> need to go shopping some stuff. will continue later. :>
[13:52] * bandrus (~brian@port-83-236-242-66.static.qsc.de) has joined #ceph
[13:56] * Wielebny (~Icedove@cl-927.waw-01.pl.sixxs.net) has joined #ceph
[13:57] * osso (~osso@sgp01-1-78-233-150-179.fbx.proxad.net) has joined #ceph
[14:05] <cetex> but it seems like ceph is trying to recover too fast actually
[14:05] <cetex> there should be throttling in there
[14:05] <cetex> it can't create 100k connections per second and just expect it to work
[14:05] <cetex> because tcp/ip isn't designed for that. in that case it should use udp instead and do it a bit differently
[14:06] <cetex> on the other hand, still broken..
[14:06] <cetex> http://www.sebastien-han.fr/blog/2015/06/23/bootstrap-your-ceph-cluster-in-docker/
[14:06] <cetex> he had it running
[14:08] * Coestar (~Izanagi@6YRAABCS9.tor-irc.dnsbl.oftc.net) has joined #ceph
[14:14] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[14:14] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[14:14] <cetex> or "isn't designed for that" is not the right answer ;)
[14:15] <cetex> but creating 100k connections during a short time interval between two hosts is messy. :)
[14:16] * C2J (~c2j@101.86.169.255) has joined #ceph
[14:17] * nardial (~ls@dslb-178-006-188-146.178.006.pools.vodafone-ip.de) Quit (Quit: Leaving)
[14:19] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[14:19] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[14:21] <cetex> and kinda guaranteeing that something will break.
[14:22] <T1> and it's a whole new cluster with no data?
[14:22] <T1> and no clients etc etc
[14:23] * linjan (~linjan@176.195.163.243) Quit (Ping timeout: 480 seconds)
[14:24] <cetex> yeah.
[14:24] <cetex> only monitors and osd's
[14:24] <cetex> clean deploy almost every time (remove logs, data directories, journals)
[14:24] <cetex> and monitor data dir
[14:24] <T1> I never saw those problems when I created a small cluster a few weeks ago
[14:25] <cetex> yeah. this is a precursor to what we're going to deploy later, so only 16 nodes (3 doubling as monitors as well)
[14:25] <cetex> but the plan is to scale it up quite a bit once we have a reproducible stable deploy
[14:26] <cetex> gotta go, will be back in a while.
[14:32] * bandrus (~brian@port-83-236-242-66.static.qsc.de) Quit (Quit: Leaving.)
[14:32] * nils_ (~nils_@doomstreet.collins.kg) Quit (Quit: This computer has gone to sleep)
[14:33] * bandrus (~brian@port-83-236-242-66.static.qsc.de) has joined #ceph
[14:33] * mykola (~Mikolaj@91.225.201.107) has joined #ceph
[14:35] * linjan (~linjan@176.195.163.243) has joined #ceph
[14:35] * olid111119 (~olid1982@p54848EBE.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[14:36] * georgem (~Adium@75-119-226-89.dsl.teksavvy.com) has joined #ceph
[14:37] * Coestar (~Izanagi@6YRAABCS9.tor-irc.dnsbl.oftc.net) Quit ()
[14:46] * shawniverson (~shawniver@208.38.236.8) has joined #ceph
[14:46] * georgem (~Adium@75-119-226-89.dsl.teksavvy.com) Quit (Quit: Leaving.)
[14:52] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[14:52] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[14:56] * ItsCriminalAFK (~Misacorp@109.201.143.40) has joined #ceph
[14:58] * Vacuum__ (~Vacuum@i59F7A2B4.versanet.de) has joined #ceph
[15:00] * danieagle (~Daniel@179.110.34.114) has joined #ceph
[15:01] * brutuscat (~brutuscat@105.34.133.37.dynamic.jazztel.es) Quit (Remote host closed the connection)
[15:02] * brutuscat (~brutuscat@105.34.133.37.dynamic.jazztel.es) has joined #ceph
[15:04] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[15:05] * Vacuum_ (~Vacuum@88.130.211.62) Quit (Ping timeout: 480 seconds)
[15:05] * diegows (~diegows@190.190.21.75) has joined #ceph
[15:09] * jdillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) has joined #ceph
[15:10] * brutuscat (~brutuscat@105.34.133.37.dynamic.jazztel.es) Quit (Ping timeout: 480 seconds)
[15:11] * yanzheng (~zhyan@171.216.95.21) Quit (Quit: This computer has gone to sleep)
[15:12] * jdillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) Quit ()
[15:17] * olid111119 (~olid1982@p54848EBE.dip0.t-ipconnect.de) has joined #ceph
[15:20] * wyang (~wyang@46.21.158.66) has joined #ceph
[15:25] * ItsCriminalAFK (~Misacorp@6YRAABCUK.tor-irc.dnsbl.oftc.net) Quit ()
[15:28] <cetex> so..
[15:36] <wyang> Hii guys, I tried to add a new 10G rbd as sdb in guest os, and enable detect zero functionality, and then execute "cat /dev/zero >/dev/sdb" in guest. But I find that amount of the object of rbd block are created by the command.
[15:40] <wyang> seems that the detect zeros fails to take effect...
[15:40] <wyang> How about the detect zero functionality on rbd?
[15:43] * nardial (~ls@dslb-178-006-188-146.178.006.pools.vodafone-ip.de) has joined #ceph
[15:55] * danieagle (~Daniel@179.110.34.114) Quit (Quit: Obrigado por Tudo! :-) inte+ :-))
[15:55] * brutuscat (~brutuscat@105.34.133.37.dynamic.jazztel.es) has joined #ceph
[15:57] * vbellur (~vijay@c-71-234-227-202.hsd1.ma.comcast.net) Quit (Ping timeout: 480 seconds)
[15:59] * TomasCZ (~TomasCZ@yes.tenlab.net) has joined #ceph
[15:59] * nardial (~ls@dslb-178-006-188-146.178.006.pools.vodafone-ip.de) Quit (Quit: Leaving)
[15:59] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[15:59] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[16:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Remote host closed the connection)
[16:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[16:02] * johnhunter (~hunter@222.29.39.73) Quit (Quit: Leaving)
[16:06] * lobstar (~Guest1390@se3x.mullvad.net) has joined #ceph
[16:15] * linjan (~linjan@176.195.163.243) Quit (Ping timeout: 480 seconds)
[16:17] * C2J (~c2j@101.86.169.255) Quit (Remote host closed the connection)
[16:17] * linjan (~linjan@176.195.163.243) has joined #ceph
[16:36] * lobstar (~Guest1390@6YRAABCVY.tor-irc.dnsbl.oftc.net) Quit ()
[16:36] <cetex> so. i've run some iperf tests, some minor packetloss on udp when i push past 1Gbit, i guess this is related to that the kernel network parameters need some tuning but as ceph doesn't use udp intensively it shouldn't be a problem.
[16:37] <cetex> throughput is decent: [SUM] 0.0-10.0 sec 21.9 GBytes 18.8 Gbits/sec
[16:37] <cetex> packetloss is 0.
[16:37] <cetex> unless you push one session past 8-9Gbit
[16:37] <cetex> and everything besides ceph works ;>
[16:38] * stj (~stj@0001c20c.user.oftc.net) Quit (Quit: reboot)
[16:40] * stj (~stj@2604:a880:800:10::2cc:b001) has joined #ceph
[16:42] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Remote host closed the connection)
[16:43] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[16:51] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Remote host closed the connection)
[16:53] * TMM (~hp@178-84-46-106.dynamic.upc.nl) Quit (Remote host closed the connection)
[16:54] * TMM (~hp@178-84-46-106.dynamic.upc.nl) has joined #ceph
[16:58] * diegows (~diegows@190.190.21.75) Quit (Ping timeout: 480 seconds)
[17:09] * chasmo77 (~chas77@158.183-62-69.ftth.swbr.surewest.net) Quit (Quit: It's just that easy)
[17:12] * shawniverson (~shawniver@208.38.236.8) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * davidz (~davidz@2605:e000:1313:8003:20f1:6dfa:24cd:5f85) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * tsg__ (~tgohad@192.55.54.40) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * shakamunyi (~shakamuny@c-67-180-191-38.hsd1.ca.comcast.net) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * magicrobotmonkey (~magicrobo@8.29.8.68) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * smerz (~ircircirc@37.74.194.90) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * trociny (~mgolub@93.183.239.2) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * loicd (~loicd@cmd179.fsffrance.org) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * jcsp (~jspray@82-71-16-249.dsl.in-addr.zen.co.uk) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * blynch (~blynch@vm-nat.msi.umn.edu) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * tnt (~tnt@ec2-54-200-98-43.us-west-2.compute.amazonaws.com) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * terje (~root@135.109.216.239) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * sw3 (sweaung@2400:6180:0:d0::66:100f) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * oblu (~o@62.109.134.112) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * beardo (~beardo__@beardo.cc.lehigh.edu) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * arbrandes (~arbrandes@ec2-54-172-54-135.compute-1.amazonaws.com) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * skullone (~skullone@shell.skull-tech.com) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * jklare (~jklare@185.27.181.36) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * hchen (~hchen@nat-pool-bos-t.redhat.com) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * acaos (~zac@209.99.103.42) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * rkeene (1011@oc9.org) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * mdxi (~mdxi@50-199-109-154-static.hfc.comcastbusiness.net) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * portante (~portante@nat-pool-bos-t.redhat.com) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * fli (fli@eastside.wirebound.net) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * ndru (~jawsome@00020819.user.oftc.net) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * fouxm (~foucault@ks01.commit.ninja) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * mfa298 (~mfa298@krikkit.yapd.net) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * destrudo (~destrudo@64.142.74.180) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * Bosse (~bosse@rifter2.klykken.com) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * thadood (~thadood@slappy.thunderbutt.org) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * chutz (~chutz@rygel.linuxfreak.ca) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * Gugge-47527 (gugge@92.246.2.105) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * Larsen (~andreas@www.larsen.pl) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * pdrakewe_ (~pdrakeweb@oh-71-50-39-25.dhcp.embarqhsd.net) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * timfreund (~tim@ec2-54-209-140-45.compute-1.amazonaws.com) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * dmsimard (~dmsimard@realm.dmsimard.com) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * ccourtaut (~ccourtaut@178.62.125.124) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * wkennington_ (~william@c-50-184-242-109.hsd1.ca.comcast.net) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * jluis (~joao@8.184.114.89.rev.vodafone.pt) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * aiicore (~aiicore@s30.linuxpl.com) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * alfredodeza (~alfredode@198.206.133.89) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * scuttlemonkey (~scuttle@nat-pool-rdu-t.redhat.com) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * andrewschoen (~andrewsch@50.56.86.195) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * joshd (~jdurgin@206.169.83.146) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * mattronix (~quassel@server1.mattronix.nl) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * phantomcircuit (~phantomci@strateman.ninja) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * mjevans (~mjevans@li984-246.members.linode.com) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * carter (~carter@li98-136.members.linode.com) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * trey (~trey@trey.user.oftc.net) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * dougf (~dougf@96-38-99-179.dhcp.jcsn.tn.charter.com) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * funnel (~funnel@0001c7d4.user.oftc.net) Quit (synthon.oftc.net beauty.oftc.net)
[17:12] * wkennington (~william@c-50-184-242-109.hsd1.ca.comcast.net) has joined #ceph
[17:16] * funnel_ (~funnel@81.4.123.134) has joined #ceph
[17:16] * shawniverson (~shawniver@208.38.236.8) has joined #ceph
[17:16] * davidz (~davidz@2605:e000:1313:8003:20f1:6dfa:24cd:5f85) has joined #ceph
[17:16] * tsg__ (~tgohad@192.55.54.40) has joined #ceph
[17:16] * shakamunyi (~shakamuny@c-67-180-191-38.hsd1.ca.comcast.net) has joined #ceph
[17:16] * magicrobotmonkey (~magicrobo@8.29.8.68) has joined #ceph
[17:16] * tnt (~tnt@ec2-54-200-98-43.us-west-2.compute.amazonaws.com) has joined #ceph
[17:16] * smerz (~ircircirc@37.74.194.90) has joined #ceph
[17:16] * trociny (~mgolub@93.183.239.2) has joined #ceph
[17:16] * loicd (~loicd@cmd179.fsffrance.org) has joined #ceph
[17:16] * jcsp (~jspray@82-71-16-249.dsl.in-addr.zen.co.uk) has joined #ceph
[17:16] * blynch (~blynch@vm-nat.msi.umn.edu) has joined #ceph
[17:16] * thadood (~thadood@slappy.thunderbutt.org) has joined #ceph
[17:16] * Bosse (~bosse@rifter2.klykken.com) has joined #ceph
[17:16] * funnel (~funnel@0001c7d4.user.oftc.net) has joined #ceph
[17:16] * alfredodeza (~alfredode@198.206.133.89) has joined #ceph
[17:16] * aiicore (~aiicore@s30.linuxpl.com) has joined #ceph
[17:16] * jluis (~joao@8.184.114.89.rev.vodafone.pt) has joined #ceph
[17:16] * destrudo (~destrudo@64.142.74.180) has joined #ceph
[17:16] * mfa298 (~mfa298@krikkit.yapd.net) has joined #ceph
[17:16] * joshd (~jdurgin@206.169.83.146) has joined #ceph
[17:16] * dougf (~dougf@96-38-99-179.dhcp.jcsn.tn.charter.com) has joined #ceph
[17:16] * pdrakewe_ (~pdrakeweb@oh-71-50-39-25.dhcp.embarqhsd.net) has joined #ceph
[17:16] * terje (~root@135.109.216.239) has joined #ceph
[17:16] * sw3 (sweaung@2400:6180:0:d0::66:100f) has joined #ceph
[17:16] * phantomcircuit (~phantomci@strateman.ninja) has joined #ceph
[17:16] * oblu (~o@62.109.134.112) has joined #ceph
[17:16] * Gugge-47527 (gugge@92.246.2.105) has joined #ceph
[17:16] * beardo (~beardo__@beardo.cc.lehigh.edu) has joined #ceph
[17:16] * timfreund (~tim@ec2-54-209-140-45.compute-1.amazonaws.com) has joined #ceph
[17:16] * arbrandes (~arbrandes@ec2-54-172-54-135.compute-1.amazonaws.com) has joined #ceph
[17:16] * mattronix (~quassel@server1.mattronix.nl) has joined #ceph
[17:16] * skullone (~skullone@shell.skull-tech.com) has joined #ceph
[17:16] * mjevans (~mjevans@li984-246.members.linode.com) has joined #ceph
[17:16] * jklare (~jklare@185.27.181.36) has joined #ceph
[17:16] * carter (~carter@li98-136.members.linode.com) has joined #ceph
[17:16] * dmsimard (~dmsimard@realm.dmsimard.com) has joined #ceph
[17:16] * hchen (~hchen@nat-pool-bos-t.redhat.com) has joined #ceph
[17:16] * scuttlemonkey (~scuttle@nat-pool-rdu-t.redhat.com) has joined #ceph
[17:16] * acaos (~zac@209.99.103.42) has joined #ceph
[17:16] * rkeene (1011@oc9.org) has joined #ceph
[17:16] * Larsen (~andreas@www.larsen.pl) has joined #ceph
[17:16] * andrewschoen (~andrewsch@50.56.86.195) has joined #ceph
[17:16] * chutz (~chutz@rygel.linuxfreak.ca) has joined #ceph
[17:16] * mdxi (~mdxi@50-199-109-154-static.hfc.comcastbusiness.net) has joined #ceph
[17:16] * portante (~portante@nat-pool-bos-t.redhat.com) has joined #ceph
[17:16] * fli (fli@eastside.wirebound.net) has joined #ceph
[17:16] * ndru (~jawsome@00020819.user.oftc.net) has joined #ceph
[17:16] * trey (~trey@trey.user.oftc.net) has joined #ceph
[17:16] * fouxm (~foucault@ks01.commit.ninja) has joined #ceph
[17:16] * ccourtaut (~ccourtaut@178.62.125.124) has joined #ceph
[17:16] * funnel (~funnel@0001c7d4.user.oftc.net) Quit (Max SendQ exceeded)
[17:16] * funnel_ is now known as funnel
[17:44] * linjan_ (~linjan@176.195.62.254) has joined #ceph
[17:47] * dgurtner (~dgurtner@217-162-119-191.dynamic.hispeed.ch) has joined #ceph
[17:48] * shaunm (~shaunm@208.102.161.229) has joined #ceph
[17:50] * TMM (~hp@178-84-46-106.dynamic.upc.nl) Quit (Remote host closed the connection)
[17:51] * linjan (~linjan@176.195.163.243) Quit (Ping timeout: 480 seconds)
[17:57] * enax (~enax@94-21-125-141.pool.digikabel.hu) has joined #ceph
[17:59] * enax (~enax@94-21-125-141.pool.digikabel.hu) has left #ceph
[18:00] * vbellur (~vijay@c-71-234-227-202.hsd1.ma.comcast.net) has joined #ceph
[18:07] * Mika_c (~Mika@36-227-14-162.dynamic-ip.hinet.net) Quit (Read error: Connection reset by peer)
[18:22] * dgurtner (~dgurtner@217-162-119-191.dynamic.hispeed.ch) Quit (Ping timeout: 480 seconds)
[18:24] * tsg__ (~tgohad@192.55.54.40) Quit (Ping timeout: 480 seconds)
[18:27] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[18:36] * PcJamesy (~BillyBobJ@6YRAABC0N.tor-irc.dnsbl.oftc.net) has joined #ceph
[18:41] * mattronix (~quassel@server1.mattronix.nl) Quit (Quit: No Ping reply in 180 seconds.)
[18:43] * mattronix (~quassel@server1.mattronix.nl) has joined #ceph
[18:45] * TMM (~hp@178-84-46-106.dynamic.upc.nl) has joined #ceph
[18:48] * EinstCrazy (~EinstCraz@117.13.201.130) Quit (Remote host closed the connection)
[18:48] * EinstCrazy (~EinstCraz@117.13.201.130) has joined #ceph
[18:48] * user1 (~user1@75-128-209-103.dhcp.trcy.mi.charter.com) has left #ceph
[18:56] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[18:56] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[18:56] * EinstCrazy (~EinstCraz@117.13.201.130) Quit (Ping timeout: 480 seconds)
[18:56] * shawniverson (~shawniver@208.38.236.8) Quit (Read error: Connection reset by peer)
[18:57] * shawniverson (~shawniver@208.38.236.8) has joined #ceph
[18:58] * moore (~moore@71-211-73-118.phnx.qwest.net) has joined #ceph
[19:02] * madkiss1 (~madkiss@2001:6f8:12c3:f00f:a10e:6b06:2cae:82f1) has joined #ceph
[19:06] * PcJamesy (~BillyBobJ@6YRAABC0N.tor-irc.dnsbl.oftc.net) Quit ()
[19:08] * madkiss (~madkiss@vpn141.sys11.net) Quit (Ping timeout: 480 seconds)
[19:14] * olid1111110 (~olid1982@aftr-185-17-206-155.dynamic.mnet-online.de) has joined #ceph
[19:15] * Hidendra (~Aethis@85.159.237.199) has joined #ceph
[19:19] * olid111119 (~olid1982@p54848EBE.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[19:29] * moore (~moore@71-211-73-118.phnx.qwest.net) Quit (Remote host closed the connection)
[19:35] * dgurtner (~dgurtner@217-162-119-191.dynamic.hispeed.ch) has joined #ceph
[19:39] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[19:39] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[19:45] * Hidendra (~Aethis@7V7AABQ28.tor-irc.dnsbl.oftc.net) Quit ()
[19:48] * jwilkins (~jowilkin@c-50-148-138-174.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[20:04] * dgurtner (~dgurtner@217-162-119-191.dynamic.hispeed.ch) Quit (Ping timeout: 480 seconds)
[20:11] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[20:11] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[20:33] * jwilkins (~jowilkin@2601:644:4000:97c0::4a04) has joined #ceph
[20:43] * aakso (aakso@hauki.tunkki.fi) Quit (Ping timeout: 480 seconds)
[20:45] * bj0rnar (~Bjornar@109.247.131.38) has joined #ceph
[20:59] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[20:59] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[21:00] * aakso (aakso@hauki.tunkki.fi) has joined #ceph
[21:14] * EinstCrazy (~EinstCraz@117.13.201.130) has joined #ceph
[21:15] * bandrus (~brian@port-83-236-242-66.static.qsc.de) Quit (Ping timeout: 480 seconds)
[21:20] * Concubidated (~Adium@pool-98-119-93-148.lsanca.fios.verizon.net) Quit (Remote host closed the connection)
[21:21] * allaok (~allaok@ARennes-658-1-52-192.w2-13.abo.wanadoo.fr) has joined #ceph
[21:22] * EinstCrazy (~EinstCraz@117.13.201.130) Quit (Ping timeout: 480 seconds)
[21:22] * KaZeR (~KaZeR@c-67-161-64-186.hsd1.ca.comcast.net) has joined #ceph
[21:26] * LeaChim (~LeaChim@host86-185-146-193.range86-185.btcentralplus.com) Quit (Ping timeout: 480 seconds)
[21:27] * bandrus (~brian@46.165.220.196) has joined #ceph
[21:28] * osso (~osso@sgp01-1-78-233-150-179.fbx.proxad.net) Quit (Quit: osso)
[21:32] * osso (~osso@sgp01-1-78-233-150-179.fbx.proxad.net) has joined #ceph
[21:36] * bandrus1 (~brian@port-83-236-242-66.static.qsc.de) has joined #ceph
[21:38] * bandrus (~brian@46.165.220.196) Quit (Ping timeout: 480 seconds)
[21:44] * rendar (~I@host112-177-dynamic.10-87-r.retail.telecomitalia.it) Quit (Ping timeout: 480 seconds)
[21:47] * rendar (~I@host112-177-dynamic.10-87-r.retail.telecomitalia.it) has joined #ceph
[21:47] * KaZeR (~KaZeR@c-67-161-64-186.hsd1.ca.comcast.net) Quit (Remote host closed the connection)
[21:53] * brianjjo (~Defaultti@ns316491.ip-37-187-129.eu) has joined #ceph
[21:55] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[21:55] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[22:16] * brutuscat (~brutuscat@105.34.133.37.dynamic.jazztel.es) Quit (Remote host closed the connection)
[22:23] * brianjjo (~Defaultti@5P6AABT65.tor-irc.dnsbl.oftc.net) Quit ()
[22:23] * w2k (~tritonx@46.166.190.131) has joined #ceph
[22:27] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[22:27] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[22:34] * mykola (~Mikolaj@91.225.201.107) Quit (Quit: away)
[22:36] * wjw-freebsd (~wjw@smtp.digiware.nl) has joined #ceph
[22:41] * bandrus1 (~brian@port-83-236-242-66.static.qsc.de) Quit (Quit: Leaving.)
[22:41] * bandrus (~brian@port-83-236-242-66.static.qsc.de) has joined #ceph
[22:49] * babilen (~babilen@babilen.user.oftc.net) Quit (Quit: leaving)
[22:50] * babilen (~babilen@babilen.user.oftc.net) has joined #ceph
[22:51] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[22:51] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[22:53] * w2k (~tritonx@4Z9AABS37.tor-irc.dnsbl.oftc.net) Quit ()
[22:58] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[22:58] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[23:02] * olid1111111 (~olid1982@aftr-185-17-204-211.dynamic.mnet-online.de) has joined #ceph
[23:04] * bandrus (~brian@port-83-236-242-66.static.qsc.de) Quit (Quit: Leaving.)
[23:05] * olid1111112 (~olid1982@aftr-185-17-206-208.dynamic.mnet-online.de) has joined #ceph
[23:06] * olid1111110 (~olid1982@aftr-185-17-206-155.dynamic.mnet-online.de) Quit (Ping timeout: 480 seconds)
[23:08] * olid1111113 (~olid1982@p54848EBE.dip0.t-ipconnect.de) has joined #ceph
[23:10] * olid1111111 (~olid1982@aftr-185-17-204-211.dynamic.mnet-online.de) Quit (Ping timeout: 480 seconds)
[23:10] * johnavp1989 (~jpetrini@pool-100-14-5-21.phlapa.fios.verizon.net) has joined #ceph
[23:13] * osso (~osso@sgp01-1-78-233-150-179.fbx.proxad.net) Quit (Quit: osso)
[23:13] * olid1111112 (~olid1982@aftr-185-17-206-208.dynamic.mnet-online.de) Quit (Ping timeout: 480 seconds)
[23:14] * rabeeh (~rabeeh@77.125.14.91) Quit (Ping timeout: 480 seconds)
[23:15] * portante (~portante@nat-pool-bos-t.redhat.com) Quit (Ping timeout: 480 seconds)
[23:23] * rabeeh (~rabeeh@77.125.14.91) has joined #ceph
[23:26] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[23:33] * shawniverson (~shawniver@208.38.236.8) Quit (Remote host closed the connection)
[23:34] * shawniverson (~shawniver@208.38.236.8) has joined #ceph
[23:35] * linjan_ (~linjan@176.195.62.254) Quit (Ping timeout: 480 seconds)
[23:52] * portante (~portante@nat-pool-bos-t.redhat.com) has joined #ceph
[23:54] <cetex> hm hm
[23:54] <cetex> maybe queueing in the nics.
[23:54] <cetex> :>
[23:56] <darkfader> cetex: i (last week) learned to find more elaborate error counters via netstat
[23:56] <darkfader> i used to know ethtool
[23:56] <darkfader> but many counters are not propagated to it and netstat -s had the more funny stuff
[23:57] <darkfader> ah and ethtool -S actually showed something for that broken box, too
[23:57] <darkfader> like this:
[23:57] <darkfader> rx_no_buffer_count: 9548
[23:57] <darkfader> rx_missed_errors: 27295
[23:57] <darkfader> that was a hardware offloading bug, don't thing it would apply to yours
[23:58] * allaok (~allaok@ARennes-658-1-52-192.w2-13.abo.wanadoo.fr) has left #ceph
[23:58] <cetex> yeah..
[23:59] <cetex> if i knew what that error message actually meant i'd be able to do a bit more about it..

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.