#ceph IRC Log

Index

IRC Log for 2015-08-21

Timestamps are in GMT/BST.

[0:02] * reed (~reed@75-101-54-18.dsl.static.fusionbroadband.com) Quit (Quit: Ex-Chat)
[0:05] * erice (~erice@75-166-50-29.hlrn.qwest.net) has joined #ceph
[0:05] * KapiteinKoffie (~ggg@marcuse-1.nos-oignons.net) has joined #ceph
[0:06] * reed (~reed@75-101-54-18.dsl.static.fusionbroadband.com) has joined #ceph
[0:09] * erice_ (~erice@50.245.231.209) has joined #ceph
[0:14] * erice (~erice@75-166-50-29.hlrn.qwest.net) Quit (Ping timeout: 480 seconds)
[0:15] * elder_ (~elder@70.96.146.23) Quit (Quit: Leaving)
[0:16] * arbrandes_ (~arbrandes@189.110.5.98) Quit (Ping timeout: 480 seconds)
[0:20] * doppelgrau (~doppelgra@pd956d116.dip0.t-ipconnect.de) has joined #ceph
[0:28] * lcurtis (~lcurtis@47.19.105.250) Quit (Remote host closed the connection)
[0:35] * KapiteinKoffie (~ggg@5NZAAGKYQ.tor-irc.dnsbl.oftc.net) Quit ()
[0:35] * zapu (~Schaap@spftor1e1.privacyfoundation.ch) has joined #ceph
[0:39] * ircolle (~Adium@2601:285:201:2bf9:79b9:3c26:8e16:15ce) Quit (Quit: Leaving.)
[0:54] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[0:54] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[1:05] * zapu (~Schaap@62SAAATU4.tor-irc.dnsbl.oftc.net) Quit ()
[1:05] * Rens2Sea1 (~Enikma@chomsky.torservers.net) has joined #ceph
[1:08] * oms101_ (~oms101@p20030057EA7E8E00EEF4BBFFFE0F7062.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[1:12] * lx0 (~aoliva@lxo.user.oftc.net) has joined #ceph
[1:12] * reset11 (~reset11@chello080108240025.2.14.vie.surfer.at) Quit (Ping timeout: 480 seconds)
[1:13] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[1:13] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[1:15] * kutija (~kutija@95.180.90.38) Quit (Quit: Textual IRC Client: www.textualapp.com)
[1:17] * oms101_ (~oms101@p20030057EA1AB400EEF4BBFFFE0F7062.dip0.t-ipconnect.de) has joined #ceph
[1:18] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[1:21] * rendar (~I@host187-231-dynamic.252-95-r.retail.telecomitalia.it) Quit ()
[1:28] * linuxkidd (~linuxkidd@41.sub-70-195-74.myvzw.com) Quit (Ping timeout: 480 seconds)
[1:28] * linuxkidd (~linuxkidd@41.sub-70-195-74.myvzw.com) has joined #ceph
[1:30] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[1:30] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[1:35] * Rens2Sea1 (~Enikma@62SAAATWB.tor-irc.dnsbl.oftc.net) Quit ()
[1:36] * Schaap (~ZombieL@1ADAAAQYT.tor-irc.dnsbl.oftc.net) has joined #ceph
[1:39] * analbeard (~shw@5.80.204.111) has joined #ceph
[1:41] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[1:41] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[1:42] * daviddcc (~dcasier@84.197.151.77.rev.sfr.net) has joined #ceph
[1:45] * angdraug (~angdraug@12.164.168.117) Quit (Quit: Leaving)
[1:46] * togdon (~togdon@c-73-25-57-104.hsd1.or.comcast.net) Quit (Quit: Bye-Bye.)
[1:48] * moore (~moore@64.202.160.88) Quit (Remote host closed the connection)
[1:52] * moore (~moore@64.202.160.88) has joined #ceph
[2:02] * daviddcc (~dcasier@84.197.151.77.rev.sfr.net) Quit (Ping timeout: 480 seconds)
[2:03] * ivotron (~ivotron@c-67-169-145-20.hsd1.ca.comcast.net) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[2:04] * fsimonce (~simon@host93-234-dynamic.252-95-r.retail.telecomitalia.it) Quit (Quit: Coyote finally caught me)
[2:05] * Schaap (~ZombieL@1ADAAAQYT.tor-irc.dnsbl.oftc.net) Quit ()
[2:05] * Freddy (~drdanick@62SAAATYJ.tor-irc.dnsbl.oftc.net) has joined #ceph
[2:06] * oms101_ (~oms101@p20030057EA1AB400EEF4BBFFFE0F7062.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[2:13] <olid1988> ceph -w
[2:13] <olid1988> ^CError connecting to cluster: InterruptedOrTimeoutError
[2:13] <olid1988> nothing in logs
[2:13] <olid1988> that all happend after i switched the public and the private network to mtu 9216
[2:13] <olid1988> jumbo frames
[2:15] * oms101_ (~oms101@p20030057EA048F00EEF4BBFFFE0F7062.dip0.t-ipconnect.de) has joined #ceph
[2:15] <olid1988> ceph -w
[2:15] <olid1988> 2015-08-21 02:14:20.542742 7f4b88b59700 0 monclient(hunting): authenticate timed out after 300
[2:15] <olid1988> 2015-08-21 02:14:20.542777 7f4b88b59700 0 librados: client.admin authentication error (110) Connection timed out
[2:15] <olid1988> Error connecting to cluster: TimedOut
[2:16] * yguang11 (~yguang11@66.228.162.44) Quit (Remote host closed the connection)
[2:18] * kevinc_ (~kevinc__@client65-64.sdsc.edu) Quit (Quit: Leaving)
[2:22] * lightspeed (~lightspee@2001:8b0:16e:1:8326:6f70:89f:8f9c) Quit (Ping timeout: 480 seconds)
[2:27] * erice (~erice@75-166-50-29.hlrn.qwest.net) has joined #ceph
[2:29] * moore (~moore@64.202.160.88) Quit (Remote host closed the connection)
[2:29] <olid1988> lol ceph can not handle mtu 9216, i have to go to mtu 9000 ^^;
[2:31] <via> what does 'ceph can not handle mut 9216' mean
[2:31] <via> more likely your switch or nic cannot handle mtu 9216
[2:31] * lightspeed (~lightspee@2001:8b0:16e:1:8326:6f70:89f:8f9c) has joined #ceph
[2:31] * alssi (~alssi@114.111.60.56) has joined #ceph
[2:32] * erice_ (~erice@50.245.231.209) Quit (Ping timeout: 480 seconds)
[2:32] <Nats_> has nothing to do with ceph, it will be a mismatch of MTU being inclusive of ethernet frame+vlan tag on one end and exclusive on the other
[2:33] <olid1988> the switch ( juniper ex2200 ) can handle it
[2:33] <olid1988> and the nic can too, its coming up, ping is working
[2:33] <via> prove it with ping of appropriate payload size
[2:33] <Nats_> hence why you'll often see switch default to 1518 or 1522 mtu while linux mtu is set to 1500
[2:35] * lucas (~Thunderbi@218.76.52.64) has joined #ceph
[2:35] * erice (~erice@75-166-50-29.hlrn.qwest.net) Quit (Ping timeout: 480 seconds)
[2:35] * doppelgrau (~doppelgra@pd956d116.dip0.t-ipconnect.de) Quit (Quit: doppelgrau)
[2:35] * Freddy (~drdanick@62SAAATYJ.tor-irc.dnsbl.oftc.net) Quit ()
[2:35] * Scrin (~jakekosbe@89.105.194.83) has joined #ceph
[2:44] <olid1988> ok, as it seems, my mistake was to switch mtu on the nic to 9216 and on the switch aswell to 9216, seems thats too much too close, the switch has to have more mtu, as it seems the nic mtu 9216 is not exactly the same like the switch mtu 9216
[2:45] * analbeard (~shw@5.80.204.111) Quit (Quit: Leaving.)
[2:46] <Nats_> typically you need to subtract 18, or 22 if you are using vlan tagging
[2:46] <olid1988> sorry, maybe its just too late for me already, not to think about such basic issues
[2:47] <via> you will gain statistically insignificant performance by using 9216 instead of 9000, its probably easier to just use a more common frame size
[2:49] * lucas1 (~Thunderbi@218.76.52.64) has joined #ceph
[2:52] <Nats_> that's also true; and you avoid problems in the future if you introduce some network gear that is 9000'ish but not exactly the same
[2:52] * jclm (~jclm@128.250.7.92) has joined #ceph
[2:56] * lucas (~Thunderbi@218.76.52.64) Quit (Ping timeout: 480 seconds)
[2:58] * dragon9783 (~jieliu@119.254.232.38) has joined #ceph
[2:59] <evilrob> so from what I'm seeing in tracker, ceph pretty much doesn't support multipath?
[3:01] <olid1988> ceph is using multiple paths ( osd's ) at the same time
[3:01] <olid1988> if you refere to iscsi, no there is no iscsi support included
[3:02] <evilrob> olid1988: I'm talking about the linux multipath driver. we're serving up LUNs from a san for OSDs with multiple paths.
[3:03] <olid1988> if you create it as rdb device, use tgt, then its working as far as i can see
[3:04] <evilrob> I think you misunderstand the side of the config I'm on. I'm adding OSDs to my ceph cluster. Not creating rbd devices being served from a ceph cluster.
[3:04] <evilrob> http://tracker.ceph.com/issues/5739
[3:05] * ivotron (~ivotron@172.56.17.181) has joined #ceph
[3:05] * Scrin (~jakekosbe@62SAAATZV.tor-irc.dnsbl.oftc.net) Quit ()
[3:05] * Esvandiary (~pakman__@tor2e1.privacyfoundation.ch) has joined #ceph
[3:06] <olid1988> i think, that this concept you refere to ( if i understand you right ) is not in harmony with ceph
[3:06] <olid1988> ceph want to manage itself where the data comes from
[3:06] <olid1988> if the hardware below ceph has its own logic and concepts to do things, i am not sure that the result will be nice
[3:07] <evilrob> multiple paths to the same disk. redundancy in access, not in data.
[3:07] * dopesong (~dopesong@lb1.mailer.data.lt) Quit (Ping timeout: 480 seconds)
[3:10] * dgurtner (~dgurtner@cpc72725-farn7-2-0-cust591.6-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[3:10] <evilrob> I'll take the feature request as evidence that this support isn't built into ceph-disk. I'll partition them myself and go from there. If I can get it to activate vs the multipath device I'll be happy. If not, I guess we can strip multipathing out of our config, but it's kind of silly given the hardwre we've got
[3:13] * zhaochao (~zhaochao@125.39.8.233) has joined #ceph
[3:14] * olid1989 (~olid1982@aftr-185-17-204-87.dynamic.mnet-online.de) has joined #ceph
[3:16] * cholcombe (~chris@c-73-180-29-35.hsd1.or.comcast.net) Quit (Ping timeout: 480 seconds)
[3:16] <badone> SAN LUNs?
[3:16] <evilrob> yes
[3:17] <evilrob> ceph-disk keeps telling me that /dev/mapper/mpath1 isn't a disk
[3:17] <badone> evilrob: Ceph was made to work with DAS or JBOD
[3:17] <evilrob> yes. I know
[3:18] <badone> I've never really understood the desire to use SAN storage for OSDs
[3:18] <evilrob> I was just hoping that it would be unix enough to work with a disk device. It's being picky about something with the mpath bits and I'm just trying to get around it.
[3:18] <via> you should use the high availablility provided by ceph, not try to make the ceph components HA
[3:18] <evilrob> storage density.
[3:19] * olid1988 (~olid1982@185.17.206.29) Quit (Ping timeout: 480 seconds)
[3:19] * ivotron (~ivotron@172.56.17.181) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[3:19] <via> anyway, ceph technically only operates on mountpoints
[3:19] <badone> In that case why use Ceph at all?
[3:19] <via> you could always mount the multipath device yourself and then set up a ceph osd on it
[3:20] * yghannam (~yghannam@0001f8aa.user.oftc.net) Quit (Ping timeout: 480 seconds)
[3:20] <evilrob> via: I'm thinking that's the workaround I'm going to go with. It shouldn't hurt in performance noticably
[3:20] <evilrob> badone: scalability
[3:21] <badone> evilrob: by adding storage?
[3:21] <evilrob> sure, I can carve out this lun and hand it to my cinder volume box.
[3:21] <evilrob> I can do that with each of my luns and cinder boxen
[3:22] <badone> sound slike you need cinder to talk directly to your SAN
[3:22] <evilrob> then when use by customers (who are unpredictable) becomes becomes lopsided, I have to mess with moving things around.
[3:23] <badone> fair enough, using Ceph to work around a deficiency in your SAN. I get it
[3:23] <evilrob> troll
[3:23] <badone> I beg your pardon?
[3:24] * rotbeard (~redbeard@cm-171-100-223-199.revip10.asianet.co.th) has joined #ceph
[3:24] <evilrob> I come here asking questions, you talk to me like I'm stupid, then tell me my system is deficient. You're clearly trolling me.
[3:24] <via> can't cinder talk ceph directly?
[3:24] <evilrob> via: yes, that's the way I'm planning on setting it up
[3:25] <evilrob> cinder will talk rbd to ceph. I just have to do the mpath bits myself.
[3:25] <via> then why do you need anything with luns or multipath
[3:25] * derjohn_mobi (~aj@x590c3423.dyn.telefonica.de) has joined #ceph
[3:25] <badone> evilrob: I think you might be a little sensitive...
[3:25] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[3:25] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Remote host closed the connection)
[3:26] <evilrob> I was hoping there was a workaround for ceph-disk not thinking these were disks. But you're right. It just makes and mounts the filesystems for me. I'll just do that myself.
[3:26] <badone> evilrob: are you saying that even if cinder could talk to your SAn directly you would still use Ceph?
[3:26] <via> evilrob: in badone's defense, it is right to question your motive here
[3:27] <via> it doesn't make a lot of sense
[3:27] <evilrob> redundancy in access paths doesn't make sense to you?
[3:27] <evilrob> it's the same disk on the other end.
[3:27] <badone> thanks via, I wondered what wrong I had done
[3:27] <via> no, it doesn't make sense with ceph
[3:27] <evilrob> I'm not trying to carve out raid luns.
[3:27] <via> losing an osd is a normal daily thing
[3:27] <badone> evilrob: I'm just going to say one more thing then leave it...
[3:28] <via> the whole point is you don't care
[3:28] <evilrob> right
[3:28] <badone> evilrob: you are looking at Ceph the wrong way
[3:28] <via> so there is no point to making it highly available
[3:28] <via> through raid or mpath
[3:28] <via> it is added complexity
[3:28] <evilrob> not using raid. mpath is a luxury that's free, no extra config required. no extra cables required.
[3:29] * TheSov2 (~TheSov@2601:246:900:8c57:d0:2c51:6c12:7bf3) has joined #ceph
[3:29] <evilrob> I understand that ceph was designed for JBOD and DAS. we just happen to have been handed 1.2P of SAN connected to a small pile of blade chassis and default setup was dual path
[3:29] <via> i don't quite follow how it has no extra config or cabling or hardware, but the point is that it buys nothing
[3:30] <via> and a SAN is logically equivilent to raid in this case
[3:30] <via> is there no way to get the san to present individual disks?
[3:30] <via> in a jbod way
[3:30] <evilrob> that is what I'm doing
[3:30] <evilrob> yes
[3:30] <evilrob> not using raid
[3:30] <via> then just use them direct
[3:31] <via> in your specific instance what are the mulltiple paths?
[3:32] <evilrob> forget it. we'll just rip out the multipath bits that are installed by default and use half the devices that present themselves.
[3:32] <evilrob> or mount the fs myself and use that
[3:32] * derjohn_mob (~aj@x590cbbeb.dyn.telefonica.de) Quit (Ping timeout: 480 seconds)
[3:33] <via> i'm actually curious as to how its providing multipath. is it separate physical networks to the san?
[3:33] <evilrob> the SAN box has multiple controllers. one path from the pair of controllers to the fabric switch.
[3:33] <via> ok
[3:33] <evilrob> the switch is connected to each blade chassis.
[3:34] <evilrob> we single pathed the connection (logically) from the SAN to the fabric and to the chassis. But the dual controllers on the storage side get us 2 paths
[3:34] <evilrob> ok, the extra config is adding one more WWN to the port group. True, that had to be done.
[3:34] <via> in this situation i suppose it makes more sense to use it, since all your 'eggs' are in one basket
[3:35] * TheSov (~TheSov@2601:246:900:8c57:d0:2c51:6c12:7bf3) Quit (Ping timeout: 480 seconds)
[3:35] <via> all osd's on one controller in one box
[3:35] <via> not really a great idea in general, but okay for proof of concept
[3:35] <evilrob> yes. in production we'll have 10 of these spread across multiple ceph nodes.
[3:35] <evilrob> again, hardware we've been handed
[3:35] * Esvandiary (~pakman__@5NZAAGK47.tor-irc.dnsbl.oftc.net) Quit ()
[3:36] * visored (~spate@176.10.99.203) has joined #ceph
[3:36] <evilrob> here in the test lab we're hooking one box up to 3 nodes.
[3:36] * dlx (~dlx@ip-62-143-166-35.hsi01.unitymediagroup.de) has joined #ceph
[3:36] <via> not ideal, but i guess if it wasn't a choice and its already bought...
[3:36] <evilrob> in prod it'll probably be 10 nodes
[3:36] <evilrob> yeah. nobody asked us. we just got handed the order sheet and told to use it.
[3:37] <via> how much bandwidth will you have to the san from the osd server?
[3:37] <evilrob> I'd love to have a pile of 2U or 4U systems full of drives, but we didn't choose :(
[3:37] <evilrob> 40Gb
[3:37] <evilrob> 4 bonded 10Gb channels
[3:38] <via> during rebuilds that might suck
[3:38] <badone> is the SAN dedicated to Ceph?
[3:39] <badone> or does Ceph have to share it?
[3:39] <evilrob> ceph only
[3:39] <evilrob> the best I can do density wise other than these SAN boxen would be one of these http://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-c240-m3-rack-server/index.html
[3:40] <evilrob> I could get a bit over 25TB out if 2U that way
[3:40] <via> that'd probably perform better
[3:40] <evilrob> yeah. given my choices, I'd rather have that.
[3:41] <via> it can take consumer sata drives, why are you limited to 25T?
[3:41] <via> oh, i guess the sff drives aren't that big yet
[3:42] <evilrob> I'll have to go this route for 2 datacenters, then we get to place our own orders (acquired startup)
[3:42] <via> i ran a very tiny cluster with 45 disks per node once
[3:43] * zhaochao (~zhaochao@125.39.8.233) Quit (Max SendQ exceeded)
[3:43] <evilrob> largest I can do in those (internal configurator) is 1.2TB SAS disks. Might be able to get larger next year when we have to do this. at least I hope so.
[3:43] <evilrob> asking for 48 of those for the next 1.2PB is going to be "interesting"
[3:46] * kefu (~kefu@114.92.110.67) has joined #ceph
[3:47] <evilrob> there are some 2TB drives out now. We should be able to get them internally next year
[3:48] * zhaochao (~zhaochao@125.39.8.233) has joined #ceph
[3:48] * dlx (~dlx@ip-62-143-166-35.hsi01.unitymediagroup.de) Quit (Quit: Leaving)
[3:54] * olid1989 (~olid1982@aftr-185-17-204-87.dynamic.mnet-online.de) Quit (Ping timeout: 480 seconds)
[3:55] * haomaiwang (~haomaiwan@60-250-10-249.HINET-IP.hinet.net) has joined #ceph
[3:59] * kefu is now known as kefu|afk
[4:00] * Mika_c (~Mika@122.146.93.152) has joined #ceph
[4:03] * kefu|afk is now known as kefu
[4:05] * visored (~spate@62SAAAT2Q.tor-irc.dnsbl.oftc.net) Quit ()
[4:05] * Sliker (~TGF@digi00411.digicube.fr) has joined #ceph
[4:07] * pdrakewe_ (~pdrakeweb@104.247.39.34) Quit (Remote host closed the connection)
[4:09] * haomaiwang (~haomaiwan@60-250-10-249.HINET-IP.hinet.net) Quit (Remote host closed the connection)
[4:10] * haomaiwang (~haomaiwan@60-250-10-249.HINET-IP.hinet.net) has joined #ceph
[4:10] * xarses_ (~xarses@12.164.168.117) Quit (Ping timeout: 480 seconds)
[4:11] * ira (~ira@c-71-233-225-22.hsd1.ma.comcast.net) Quit (Ping timeout: 480 seconds)
[4:13] <evilrob> badone: sorry about the above. been working too much and yes, probably being over-sensative (bitchy).
[4:14] <evilrob> via: thank you for making me realize I can just mount that shit myself. I knew I could do that, I just couldn't shift focus away from making ceph-disk do that for me.
[4:14] <badone> evilrob: no problem at all. I come across badly sometimes but I am what I am. no harm done :)
[4:14] <evilrob> yes... now I have whiskey
[4:14] <badone> :)
[4:21] * sankarshan (~sankarsha@121.244.87.117) has joined #ceph
[4:21] * flisky (~Thunderbi@36.110.40.22) has joined #ceph
[4:28] * steveeJ_ (~junky@HSI-KBW-46-223-54-130.hsi.kabel-badenwuerttemberg.de) has joined #ceph
[4:28] * steveeJ_ (~junky@HSI-KBW-46-223-54-130.hsi.kabel-badenwuerttemberg.de) Quit (Remote host closed the connection)
[4:31] * shyu (~Shanzhi@119.254.120.66) has joined #ceph
[4:33] * mookins (~mookins@induct3.lnk.telstra.net) Quit ()
[4:35] * Sliker (~TGF@1ADAAAQ3T.tor-irc.dnsbl.oftc.net) Quit ()
[4:36] * blank (~Kakeru@89.105.194.90) has joined #ceph
[4:36] * xarses_ (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) has joined #ceph
[4:39] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[4:39] * zack_dolby (~textual@pa3b3a1.tokynt01.ap.so-net.ne.jp) Quit (Read error: Connection reset by peer)
[4:39] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[4:40] * zack_dolby (~textual@pa3b3a1.tokynt01.ap.so-net.ne.jp) has joined #ceph
[4:44] <janos> mmm whiskey. what kind?
[4:47] * haomaiwang (~haomaiwan@60-250-10-249.HINET-IP.hinet.net) Quit (Remote host closed the connection)
[4:47] * haomaiwang (~haomaiwan@li390-182.members.linode.com) has joined #ceph
[4:48] <evilrob> lagavulin.
[4:48] <evilrob> smokey peatey goodness
[4:48] <janos> sounds like scotch whiskey to me
[4:49] <evilrob> that it is
[4:49] <evilrob> I chose the less specific noun
[4:49] <janos> my father only drinks scotch - that sounds familar
[4:49] <janos> np?
[4:49] <janos> err
[4:49] <janos> -?
[4:49] <janos> i've been enjoying some woodford reserve rye
[4:50] <evilrob> that's tasty. I saw it on a list in some magazine a while back and ordered a glass
[4:50] <janos> it's become my new favorite rye
[4:50] <janos> has a much more mature set to it than some others i enjoy
[4:51] <janos> i'd be hard-pressed to turn any down though :)
[4:54] * jclm (~jclm@128.250.7.92) Quit (Quit: Leaving.)
[4:55] <evilrob> oh yeah... setting up a test lab. size 2 is fine. I'll go to 3 in prod
[4:57] * lathiat (~lathiat@2403:dc00::117) Quit (Ping timeout: 480 seconds)
[4:58] * haomaiwang (~haomaiwan@li390-182.members.linode.com) Quit (Remote host closed the connection)
[4:58] * haomaiwang (~haomaiwan@li817-120.members.linode.com) has joined #ceph
[5:01] * haomaiwang (~haomaiwan@li817-120.members.linode.com) Quit (Remote host closed the connection)
[5:01] * haomaiwang (~haomaiwan@li1072-91.members.linode.com) has joined #ceph
[5:03] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[5:03] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[5:04] * dyasny (~dyasny@198.251.59.70) Quit (Ping timeout: 480 seconds)
[5:05] * blank (~Kakeru@1ADAAAQ44.tor-irc.dnsbl.oftc.net) Quit ()
[5:05] * dusti (~eXeler0n@exit1.ipredator.se) has joined #ceph
[5:09] <evilrob> wow... how picky is it with clock skew?
[5:09] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[5:09] <evilrob> all 3 nodes are pointed at the same ntp server, and I have offsets ranging from -0.104ms to -0.024ms
[5:10] * yanzheng (~zhyan@125.71.106.63) has joined #ceph
[5:12] * lx0 (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[5:17] * Kupo1 (~tyler.wil@23.111.254.159) Quit (Read error: Connection reset by peer)
[5:18] * haomaiwang (~haomaiwan@li1072-91.members.linode.com) Quit (Remote host closed the connection)
[5:18] * haomaiwang (~haomaiwan@li1072-91.members.linode.com) has joined #ceph
[5:23] * pdrakeweb (~pdrakeweb@173-9-49-67-NewEngland.hfc.comcastbusiness.net) has joined #ceph
[5:23] <Mika_c> Try to write into hardware clock? Or replaced ntp to "chrony".
[5:24] <evilrob> for some reason it just took a while to decide there wasn't a skew
[5:24] <evilrob> the offsets are about the same now but it's all ok
[5:27] <Kingrat> evilrob, i would suggest you run ntp locally, have a couple servers on your network peer with each other and also with internet peers
[5:28] * yguang11 (~yguang11@c-50-131-146-113.hsd1.ca.comcast.net) has joined #ceph
[5:28] <Kingrat> that is the setup i run, even better if i had a larger setup would be gps hardware
[5:28] <evilrob> Kingrat: currently it's a single NTP box in the lab. The plan is to have all the ceph nodes peer with each other and also with a pair of in lab NTP boxen that are pointed at the outside world
[5:29] * erice (~erice@c-76-120-53-165.hsd1.co.comcast.net) has joined #ceph
[5:29] <Kingrat> you probably dont want to "peer" them between many servers, i peer between 2 which keeps them in sync, then the others all sync off of either of those
[5:35] * dusti (~eXeler0n@62SAAAT52.tor-irc.dnsbl.oftc.net) Quit ()
[5:35] * BillyBobJohn (~hassifa@195-154-15-227.rev.poneytelecom.eu) has joined #ceph
[5:36] * marrusl (~mark@cpe-67-247-9-253.nyc.res.rr.com) Quit (Quit: bye!)
[5:41] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[5:41] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[5:43] * Vacuum__ (~Vacuum@88.130.214.44) has joined #ceph
[5:43] * huats (~quassel@stuart.objectif-libre.com) Quit (Ping timeout: 480 seconds)
[5:44] * kefu (~kefu@114.92.110.67) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[5:49] * Vacuum_ (~Vacuum@i59F79DAD.versanet.de) Quit (Ping timeout: 480 seconds)
[6:05] * BillyBobJohn (~hassifa@62SAAAT7I.tor-irc.dnsbl.oftc.net) Quit ()
[6:06] * dotblank (~cyphase@176.10.99.206) has joined #ceph
[6:10] * ivotron (~ivotron@c-67-169-145-20.hsd1.ca.comcast.net) has joined #ceph
[6:13] * npcomp (~npcomp@c-24-126-240-124.hsd1.ga.comcast.net) has joined #ceph
[6:18] * yguang11_ (~yguang11@2001:4998:effd:7801::10cf) has joined #ceph
[6:18] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[6:18] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[6:25] * yguang11 (~yguang11@c-50-131-146-113.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[6:26] * yguang11_ (~yguang11@2001:4998:effd:7801::10cf) Quit (Remote host closed the connection)
[6:27] * yguang11 (~yguang11@2001:4998:effd:7801::10cf) has joined #ceph
[6:32] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[6:32] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[6:35] * dotblank (~cyphase@62SAAAT8E.tor-irc.dnsbl.oftc.net) Quit ()
[6:35] * DougalJacobs (~nupanick@spftor1e1.privacyfoundation.ch) has joined #ceph
[6:36] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[6:37] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[6:38] * pdrakeweb (~pdrakeweb@173-9-49-67-NewEngland.hfc.comcastbusiness.net) Quit (Remote host closed the connection)
[6:44] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[6:44] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[6:56] * oblu (~o@62.109.134.112) Quit (Ping timeout: 480 seconds)
[7:00] * kefu (~kefu@114.92.110.67) has joined #ceph
[7:05] * DougalJacobs (~nupanick@62SAAAT88.tor-irc.dnsbl.oftc.net) Quit ()
[7:05] * W|ldCraze (~Snowman@marylou.nos-oignons.net) has joined #ceph
[7:22] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[7:22] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[7:25] * flisky (~Thunderbi@36.110.40.22) Quit (Remote host closed the connection)
[7:25] * flisky (~Thunderbi@36.110.40.30) has joined #ceph
[7:28] * flisky (~Thunderbi@36.110.40.30) Quit (Remote host closed the connection)
[7:28] * flisky (~Thunderbi@36.110.40.30) has joined #ceph
[7:30] * CheKoLyN (~saguilar@bender.parc.xerox.com) Quit (Quit: Leaving)
[7:31] * flisky1 (~Thunderbi@36.110.40.29) has joined #ceph
[7:31] * flisky (~Thunderbi@36.110.40.30) Quit (Remote host closed the connection)
[7:33] * karnan (~karnan@106.51.232.229) has joined #ceph
[7:33] * flisky1 (~Thunderbi@36.110.40.29) Quit (Remote host closed the connection)
[7:34] * flisky (~Thunderbi@36.110.40.20) has joined #ceph
[7:34] <YmrDtnJu> hi. i have the journal file on tmpfs but "ceph tell osd.0 bench" is very slow. if i put the journal on /dev/ram0, i get the full performance of the ssd that osd is on. why is tmpfs so slow? i also tried with ramfs. same problem.
[7:35] * W|ldCraze (~Snowman@1ADAAARAH.tor-irc.dnsbl.oftc.net) Quit ()
[7:36] * zc00gii (~Kyso@6.tor.exit.babylon.network) has joined #ceph
[7:44] <Nats_> the journal needs to be persistent, so turn back now
[7:44] * reed (~reed@75-101-54-18.dsl.static.fusionbroadband.com) Quit (Quit: Ex-Chat)
[7:46] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[7:46] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[7:47] <YmrDtnJu> that is not the answer to my question. :-)
[7:47] <Nats_> you're in the wrong channel if you want to know why tmpfs is slow
[7:48] * wschulze (~wschulze@cpe-69-206-242-167.nyc.res.rr.com) Quit (Quit: Leaving.)
[7:48] <YmrDtnJu> why? this is the ceph channel, isn't it?
[7:48] <Nats_> tmpfs is a standard part of linux
[7:48] <YmrDtnJu> tmpfs is very fast. :-) its not a problem with tmpfs.
[7:49] <YmrDtnJu> somehow its a problem with ceph using tmpfs. any maybe someone here has an idea.
[7:50] * kefu (~kefu@114.92.110.67) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[7:50] <YmrDtnJu> the problem is, that rbd is very very slow here. and i actually don't know why. rbd bench is fast. but kernel rbd block device are very slow and the rbd code in qemu is very slow either. i only get about 40 mb/s on a gigabit network.
[7:50] <YmrDtnJu> and i try to find the bottleneck.
[7:55] * kefu (~kefu@114.92.110.67) has joined #ceph
[7:56] <YmrDtnJu> i expect to get more than 100 mb/s on a 1 gigabit network. :-(
[7:58] * snakamoto (~Adium@192.16.26.2) Quit (Quit: Leaving.)
[8:00] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[8:00] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[8:01] * zaitcev (~zaitcev@2001:558:6001:10:61d7:f51f:def8:4b0f) Quit (Quit: Bye)
[8:01] <dragon9783> YmrDtnJu: i remember there is a keynote about ceph optimization on slideshare
[8:02] <YmrDtnJu> dragon9783: i have searched the internet for the last weeks. i guess i already read everything that can be found.
[8:03] <YmrDtnJu> it seems to me, that ceph is just slow and needs a lot of work to enhance performance. our production system will have 10g links, but we could just use 1g as it won't make any difference.
[8:03] * rotbeard (~redbeard@cm-171-100-223-199.revip10.asianet.co.th) Quit (Ping timeout: 480 seconds)
[8:03] <YmrDtnJu> fio only does about 7 mb/s :-( with osds on a 250mb/s ssd and journal in ram!
[8:05] * zc00gii (~Kyso@1ADAAARBH.tor-irc.dnsbl.oftc.net) Quit ()
[8:05] * w2k (~Throlkim@5NZAAGLD7.tor-irc.dnsbl.oftc.net) has joined #ceph
[8:06] <YmrDtnJu> rbd and qemu have greate features. snapshots are create, thin provisioning. but performance is very very bad.
[8:06] <Nats_> its slow but its not that slow
[8:06] <dragon9783> YmrDtnJu: such as keep FD, sparse read/write, drop default limits , use rbd cache etc
[8:07] * shylesh__ (~shylesh@121.244.87.118) has joined #ceph
[8:07] <dragon9783> you can read it , named <<build an high-performance and hight-durable block storage>>
[8:07] * kefu (~kefu@114.92.110.67) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[8:07] <YmrDtnJu> dragon9783: i have already tried everything i found. :-)
[8:09] * flisky (~Thunderbi@36.110.40.20) Quit (Quit: flisky)
[8:10] * zack_dolby (~textual@pa3b3a1.tokynt01.ap.so-net.ne.jp) Quit (Quit: My MacBook has gone to sleep. ZZZzzz???)
[8:10] * zack_dolby (~textual@pa3b3a1.tokynt01.ap.so-net.ne.jp) has joined #ceph
[8:14] * yguang11 (~yguang11@2001:4998:effd:7801::10cf) Quit ()
[8:18] * zack_dolby (~textual@pa3b3a1.tokynt01.ap.so-net.ne.jp) Quit (Ping timeout: 480 seconds)
[8:18] <rkeene> It's definitely slow
[8:19] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[8:19] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[8:19] * oblu (~o@62.109.134.112) has joined #ceph
[8:22] * vikhyat (~vumrao@121.244.87.116) has joined #ceph
[8:23] <YmrDtnJu> but i don't know why.
[8:23] * rdas (~rdas@121.244.87.116) has joined #ceph
[8:23] <as0bu> YmrDtnJu what are you system specs?
[8:23] <as0bu> for your OSDs and mons
[8:24] <YmrDtnJu> it is a testsystem. i have three pcs. each has 16 gib of ram, 4 core with ht intel core i7. two network cards on two cisco switch, each 1 gbit/s. i did tests with iperf. i get the full performance on the network.
[8:25] <YmrDtnJu> each pc has one mon and two osds. one osd on the ssd with 250mb/s and one a hdd with 40 mb/s.
[8:26] <as0bu> How big is the SSD and HDD?
[8:26] <YmrDtnJu> there is one pool with a cache over the ssd osds.
[8:26] <YmrDtnJu> the ssd is 250 gb and hdd is 750 gb.
[8:26] <as0bu> Where are you journaling?
[8:26] <YmrDtnJu> in ram.
[8:26] <rkeene> Why does Ceph report clock skew between monitors when none exists ?
[8:27] <YmrDtnJu> there is a ram disk of 8 gb size for every osd.
[8:27] <as0bu> So 16GB total?
[8:27] <YmrDtnJu> as0bu: the filesystem for all osds is btrfs.
[8:27] <YmrDtnJu> as0bu: yes.
[8:28] <as0bu> and the RAM in each server is also 16GB?
[8:29] <YmrDtnJu> as0bu: yes.
[8:29] <as0bu> Have you tried journaling on the SSD with only one OSD?
[8:29] <YmrDtnJu> as0bu: yes. that did note make a difference.
[8:29] * janos_ (~messy@static-71-176-211-4.rcmdva.fios.verizon.net) has joined #ceph
[8:30] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[8:30] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[8:30] <as0bu> Ubuntu?
[8:30] <YmrDtnJu> debian wheezy.
[8:30] <YmrDtnJu> ceph 0.94.2
[8:32] <as0bu> Where are you mounting the RBD?
[8:32] <YmrDtnJu> on a fourth pc connected with the public network of the ceph cluster. same machine as the ceph nodes.
[8:32] <YmrDtnJu> qemu version is 2.3
[8:33] <YmrDtnJu> with virtio-scsi as the virtual block device and rbd as the backend.
[8:33] * karnan (~karnan@106.51.232.229) Quit (Ping timeout: 480 seconds)
[8:33] <as0bu> when you mount the rbd on the host machine do you still see poor performance?
[8:34] <YmrDtnJu> the docs say, that i should not do that. so i did not try. :-)
[8:34] <YmrDtnJu> i can try now.
[8:34] <rkeene> Is there any way to convince Ceph to check again regarding clock skew ? It's reporting HEALTH_WARN and indicating clock skew where none exists.
[8:35] * yanzheng (~zhyan@125.71.106.63) Quit (Ping timeout: 480 seconds)
[8:35] * w2k (~Throlkim@5NZAAGLD7.tor-irc.dnsbl.oftc.net) Quit ()
[8:36] <as0bu> rkeene are you using NTP?
[8:36] * Zeis (~Teddybare@torland1-this.is.a.tor.exit.server.torland.is) has joined #ceph
[8:36] <YmrDtnJu> as0bu: a second please.
[8:36] <as0bu> YmrDtnJu Don't worry I won't forget about you
[8:36] <rkeene> as0bu, Yes.
[8:36] <as0bu> You're first and I'm sure that rkeene understands
[8:36] * janos (~messy@static-71-176-211-4.rcmdva.fios.verizon.net) Quit (Ping timeout: 480 seconds)
[8:37] <as0bu> rkeene does it tell you which mon?
[8:38] <rkeene> Yes, clock skew detected on mon.storage-3
[8:39] <rkeene> 3 mons, time on each are (1,2,3): 1440138467.2506216 1440138467.2497072 1440138467.250125
[8:39] * yanzheng (~zhyan@182.139.20.109) has joined #ceph
[8:39] <Mika_c> Hi, have you use "ceph health detail " to check? Default the monitors will allow clocks to drift up to 0.05 seconds.
[8:40] * Hemanth (~Hemanth@121.244.87.117) has joined #ceph
[8:40] <as0bu> Yeah it should tell you which one is having problems and how far off it is from the maximum
[8:40] <as0bu> YmrDtnJu how are you doing over there?
[8:40] <rkeene> mon.storage-3 addr 10.0.0.3:6789/0 clock skew 0.566944s > max 0.05s (latency 0.107542s)
[8:41] * rdas (~rdas@121.244.87.116) Quit (Quit: Leaving)
[8:41] <YmrDtnJu> as0bu: i am done. about 30mb/s read and 10 mb/s write.
[8:41] <rkeene> Which is wrong (it could have been wrong at one point, since the machines just came back up)
[8:41] <as0bu> MB or mb?
[8:41] <rkeene> How do I get it to check again ?
[8:42] <YmrDtnJu> as0bu: megabytes
[8:42] * samw (~sam@94-224-154-150.access.telenet.be) has joined #ceph
[8:42] <as0bu> YmrDtnJu are you using two interfaces?
[8:42] <as0bu> I think you said you were
[8:42] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[8:42] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[8:43] <as0bu> with two separate switches even correct?
[8:43] <YmrDtnJu> as0bu: yes and yes.
[8:43] <YmrDtnJu> the switches are cisco 8 pot 2960g.
[8:43] <YmrDtnJu> port
[8:43] <as0bu> Are you interfaces clean? i.e. not dropped packets
[8:43] <YmrDtnJu> as0bu: yes, clean
[8:45] <as0bu> I'm not totally sure if this is your issue or not but the fact that you have 16GB of journaling in our RAM, on a 16GB RAM system is troubling.
[8:45] <Mika_c> rkeene, Maybe you need synchronize time first.
[8:45] <as0bu> rkeene have you restarted the mon service that's reporting the skew?
[8:46] * karnan (~karnan@106.216.164.93) has joined #ceph
[8:46] <as0bu> YmrDtnJu the reason being is that the journal file gets allocated
[8:46] <YmrDtnJu> as0bu: i can try with 4 gb ram disk size.
[8:46] <YmrDtnJu> but i really don't expect that to have any difference.
[8:46] <as0bu> If you don't mind
[8:46] <YmrDtnJu> ok
[8:47] <as0bu> I'm not totally sure that's your issue, but it could be later
[8:47] <YmrDtnJu> will take some time, because i have to restart all the ceph nodes.
[8:47] <as0bu> I'll be on at least another 30min
[8:47] <YmrDtnJu> by the way: fio on the virtual machine on the other pc tells me 20 mb/s read and ybout 7 mb/s write.
[8:48] <YmrDtnJu> and 7 mb/s is realy not very fast. :-(
[8:48] <as0bu> So about the same as the host system
[8:48] <YmrDtnJu> yes
[8:48] <as0bu> Do you have swap?
[8:48] <YmrDtnJu> no
[8:48] <YmrDtnJu> fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4K --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
[8:49] <Nats_> one would not typically look at the mb/sec for rand i/o tests
[8:49] <Nats_> instead you would lookat iops
[8:49] * kefu (~kefu@114.92.110.67) has joined #ceph
[8:50] <YmrDtnJu> 4976 read and 1659 write
[8:51] <YmrDtnJu> on the virtual machine.
[8:52] * dopesong (~dopesong@lb1.mailer.data.lt) has joined #ceph
[8:52] <as0bu> YmrDtnJu have you done a dd with dsync?
[8:52] <as0bu> just to see what you get
[8:52] * b0e (~aledermue@213.95.25.82) has joined #ceph
[8:52] <YmrDtnJu> as0bu: yes. dd if=/dev/zero of=file count=1024 bs=$(( 1024 * 1024 )) oflag=direct
[8:53] <YmrDtnJu> 40 mb/s on the virtual machine.
[8:53] <Nats_> thats pretty much what i'd expect with ceph for random 4k blocks across 6 osd
[8:54] <rkeene> Is this compute clock skew computed as some kind of running average ? It seems to be going down, slowly, each time it checks (apparently 5 minutes)
[8:54] <YmrDtnJu> as0bu: if i use the local ssd with qemu i get about 200 mb/s. so, qemu and virtio-scsi should not be the bottleneck.
[8:55] <YmrDtnJu> Nats_: only 3 osds. i have one pool with the three osds on hdd and one pool with the three osds on ssd. the ssd pool is a cache pool for the hdd pool.
[8:55] <YmrDtnJu> the hdd osds are never used. everything is only in the cache pool on the ssds.
[9:01] <as0bu> YmrDtnJu dd if=/dev/zero of=file count=1 bs=1G oflag=direct, see if that is better or not
[9:02] <YmrDtnJu> as0bu: i am restraring the nodes. i will try that as soon as all the nodes are back again.
[9:02] <as0bu> ok
[9:05] * Zeis (~Teddybare@1ADAAARED.tor-irc.dnsbl.oftc.net) Quit ()
[9:05] * aldiyen (~Bwana@tor-exit.15-cloud.fr) has joined #ceph
[9:10] * rdas (~rdas@121.244.87.116) has joined #ceph
[9:12] * jasuarez (~jasuarez@78.Red-83-37-130.dynamicIP.rima-tde.net) has joined #ceph
[9:13] <YmrDtnJu> as0bu: it will take time. there is another problem now.
[9:13] <as0bu> Oh ok
[9:13] <YmrDtnJu> "256 pgs: 181 peering, 31 active+clean, 44 remapped+peering"
[9:13] <YmrDtnJu> "225 pgs peering; 198 pgs stuck inactive; 225 pgs stuck unclean;"
[9:13] <YmrDtnJu> and i don't get 256 active+clean anymore.
[9:14] <YmrDtnJu> osd.5 10.7.78.63:6802/3202 failed (188 reports from 2 peers after 495.629338 >= grace 495.463968)
[9:15] <as0bu> Are all your OSDs up and in?
[9:15] <YmrDtnJu> all up and in.
[9:16] * linjan_ (~linjan@213.80.239.50) has joined #ceph
[9:17] <YmrDtnJu> only 4 up now.
[9:18] <as0bu> Did you build this by hand or using ceph-deploy?
[9:18] <YmrDtnJu> by hand
[9:18] <YmrDtnJu> seems to me, that the osds fail to communicate with each other.
[9:21] <as0bu> what is your osd_journal_size set to now?
[9:22] <YmrDtnJu> i did not set the journal size.
[9:22] <YmrDtnJu> as0bu: the network is down. i don't know why. i have to fix that.
[9:22] <as0bu> Ah ok
[9:22] <as0bu> Sorry to bail on you, but I need to get to bed
[9:23] <YmrDtnJu> as0bu: no problem. :-) thx for the help.
[9:23] <as0bu> eh I wasn't much help lol
[9:24] <YmrDtnJu> :-)
[9:24] * olid1989 (~olid1982@aftr-185-17-204-87.dynamic.mnet-online.de) has joined #ceph
[9:25] <as0bu> I'm not sure if you read any white papers on ceph benchmarks, but that would be a good place to see what to expect
[9:27] <as0bu> YmrDtnJu Here is one I think that will help you out http://www.mellanox.com/pdf/whitepapers/WP_Deploying_Ceph_over_High_Performance_Networks.pdf
[9:27] <YmrDtnJu> as0bu: i have the read first 20 pages google presented me. :-)
[9:27] <as0bu> Appendix A specifically
[9:29] <YmrDtnJu> as0bu: i know that pdf. but it does not really help me. i don't even saturate 1 gbit/s. :-)
[9:29] <as0bu> You might with larger block sizes
[9:30] <as0bu> ok, I'm out for real this time
[9:30] <YmrDtnJu> ok
[9:32] * T1w (~jens@node3.survey-it.dk) has joined #ceph
[9:32] * analbeard (~shw@support.memset.com) has joined #ceph
[9:34] * Peltzi (peltzi@peltzi.fi) has joined #ceph
[9:34] * bitserker (~toni@47.61.240.151) has joined #ceph
[9:34] * kawa2014 (~kawa@89.184.114.246) has joined #ceph
[9:35] * aldiyen (~Bwana@5NZAAGLGH.tor-irc.dnsbl.oftc.net) Quit ()
[9:36] * QuantumBeep (~colde@213-21-123-232.customer.t3.se) has joined #ceph
[9:36] * bitserker (~toni@47.61.240.151) Quit ()
[9:36] * bitserker (~toni@47.61.240.151) has joined #ceph
[9:36] * bitserker (~toni@47.61.240.151) Quit ()
[9:36] * bitserker (~toni@47.61.240.151) has joined #ceph
[9:41] * doppelgrau (~doppelgra@pd956d116.dip0.t-ipconnect.de) has joined #ceph
[9:49] * haomaiwang (~haomaiwan@li1072-91.members.linode.com) Quit (Remote host closed the connection)
[9:49] * haomaiwang (~haomaiwan@li817-120.members.linode.com) has joined #ceph
[9:52] * T1w (~jens@node3.survey-it.dk) Quit (Quit: Leaving)
[9:56] * Concubidated (~Adium@71.21.5.251) Quit (Quit: Leaving.)
[9:59] * lucas1 (~Thunderbi@218.76.52.64) Quit (Remote host closed the connection)
[9:59] * lucas (~Thunderbi@218.76.52.64) has joined #ceph
[10:00] * reset11 (~reset11@141.244.133.89) has joined #ceph
[10:01] * haomaiwang (~haomaiwan@li817-120.members.linode.com) Quit (Remote host closed the connection)
[10:02] * haomaiwang (~haomaiwan@li817-120.members.linode.com) has joined #ceph
[10:04] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[10:04] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[10:04] * jordanP (~jordan@scality-jouf-2-194.fib.nerim.net) has joined #ceph
[10:04] * huats (~quassel@stuart.objectif-libre.com) has joined #ceph
[10:05] * QuantumBeep (~colde@1ADAAARG8.tor-irc.dnsbl.oftc.net) Quit ()
[10:06] * Quatroking1 (~JWilbur@madiba.guilhem.org) has joined #ceph
[10:09] * fdmanana (~fdmanana@2001:8a0:6dfd:6d01:a859:709b:67fc:b0e2) has joined #ceph
[10:13] * T1w (~jens@node3.survey-it.dk) has joined #ceph
[10:15] * TMM (~hp@sams-office-nat.tomtomgroup.com) has joined #ceph
[10:16] * branto (~branto@178-253-131-8.3pp.slovanet.sk) has joined #ceph
[10:22] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[10:22] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[10:26] * zack_dolby (~textual@pw126236141074.12.panda-world.ne.jp) has joined #ceph
[10:27] * dgurtner (~dgurtner@149.11.161.243) has joined #ceph
[10:27] * brutuscat (~brutuscat@234.Red-79-151-98.dynamicIP.rima-tde.net) has joined #ceph
[10:28] * zack_dolby (~textual@pw126236141074.12.panda-world.ne.jp) Quit ()
[10:35] * Quatroking1 (~JWilbur@5NZAAGLIL.tor-irc.dnsbl.oftc.net) Quit ()
[10:35] * Diablodoct0r (~darkid@104.238.169.78) has joined #ceph
[10:39] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[10:40] * ngoswami (~ngoswami@121.244.87.116) has joined #ceph
[10:43] * rendar (~I@host16-175-dynamic.7-87-r.retail.telecomitalia.it) has joined #ceph
[10:46] * doppelgrau (~doppelgra@pd956d116.dip0.t-ipconnect.de) Quit (Quit: doppelgrau)
[10:54] * rotbeard (~redbeard@cm-171-100-223-199.revip10.asianet.co.th) has joined #ceph
[10:58] * ngoswami_ (~ngoswami@121.244.87.124) has joined #ceph
[11:01] * haomaiwang (~haomaiwan@li817-120.members.linode.com) Quit (Remote host closed the connection)
[11:01] * badone (~brad@CPE-121-215-241-179.static.qld.bigpond.net.au) Quit (Ping timeout: 480 seconds)
[11:01] * kawa2014 (~kawa@89.184.114.246) Quit (Ping timeout: 480 seconds)
[11:02] * haomaiwang (~haomaiwan@li817-120.members.linode.com) has joined #ceph
[11:04] * ngoswami (~ngoswami@121.244.87.116) Quit (Ping timeout: 480 seconds)
[11:05] * shylesh__ (~shylesh@121.244.87.118) Quit (Ping timeout: 480 seconds)
[11:05] * Diablodoct0r (~darkid@104.238.169.78) Quit ()
[11:05] * shishi (~datagutt@ns323918.ip-94-23-30.eu) has joined #ceph
[11:08] * tnt (~tnt@ec2-54-200-98-43.us-west-2.compute.amazonaws.com) has left #ceph
[11:13] * kawa2014 (~kawa@89.184.114.246) has joined #ceph
[11:19] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[11:19] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[11:20] <samw> Hi
[11:20] <samw> we have an increasing amount of "heartbeat_map is_healthy 'RGWProcess::m_tp thread 0x7f38c77e6700' had timed out after 600" messages in our radosgw logs
[11:20] <samw> eventually causing the radosgw to stall
[11:21] <samw> "ceph daemon /var/run/ceph/ceph-client.*.asok objecter_requests" shows "call rgw.bucket_list" ops
[11:21] <samw> ceph version is hammer 0.94.2
[11:22] <samw> anyone any idea how to get rid of those requests? restarting radosgw doesn't help
[11:22] <samw> restarting the osd neither
[11:26] * vikhyat (~vumrao@121.244.87.116) Quit (Quit: Leaving)
[11:35] * shishi (~datagutt@62SAAAUK4.tor-irc.dnsbl.oftc.net) Quit ()
[11:35] * Miho (~Xerati@marylou.nos-oignons.net) has joined #ceph
[11:36] * ngoswami_ (~ngoswami@121.244.87.124) Quit (Ping timeout: 480 seconds)
[11:38] * Sysadmin88 (~IceChat77@2.125.96.238) Quit (Quit: Few women admit their age. Few men act theirs.)
[11:39] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[11:39] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[11:42] * derjohn_mobi (~aj@x590c3423.dyn.telefonica.de) Quit (Ping timeout: 480 seconds)
[11:44] * ngoswami_ (~ngoswami@121.244.87.116) has joined #ceph
[11:45] * dragon9783 (~jieliu@119.254.232.38) Quit (Remote host closed the connection)
[11:52] * lucas (~Thunderbi@218.76.52.64) Quit (Quit: lucas)
[11:57] * rendar (~I@host16-175-dynamic.7-87-r.retail.telecomitalia.it) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * dgurtner (~dgurtner@149.11.161.243) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * jordanP (~jordan@scality-jouf-2-194.fib.nerim.net) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * linjan_ (~linjan@213.80.239.50) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * karnan (~karnan@106.216.164.93) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * Hemanth (~Hemanth@121.244.87.117) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * ivotron (~ivotron@c-67-169-145-20.hsd1.ca.comcast.net) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * lxo (~aoliva@lxo.user.oftc.net) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * owasserm (~owasserm@52D9864F.cm-11-1c.dynamic.ziggo.nl) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * jwilkins (~jowilkin@c-67-180-123-48.hsd1.ca.comcast.net) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * jdillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * bkopilov (~bkopilov@bzq-109-66-56-13.red.bezeqint.net) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * `10 (~10@69.169.91.14) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * aarontc (~aarontc@2001:470:e893::1:1) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * c4tech (~c4tech@199.91.185.156) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * elder (~elder@c-24-245-18-91.hsd1.mn.comcast.net) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * jpieper (~josh@209-6-39-224.c3-0.smr-ubr2.sbo-smr.ma.cable.rcn.com) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * Jahkeup (Jahkeup@gemini.ca.us.panicbnc.net) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * _robbat21irssi (nobody@www2.orbis-terrarum.net) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * jklare (~jklare@185.27.181.36) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * championofcyrodi (~championo@50-205-35-98-static.hfc.comcastbusiness.net) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * geli (~geli@geli-2015.its.utas.edu.au) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * mattt (~mattt@lnx1.defunct.ca) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * epf (epf@155.254.32.82) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * Kingrat (~shiny@2605:a000:161a:c022:2d0e:daff:3f4a:6f64) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * bilco105_ (~bilco105@irc.bilco105.com) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * hchen (~hchen@nat-pool-bos-t.redhat.com) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * Pintomatic (sid25118@id-25118.ealing.irccloud.com) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * Karcaw (~evan@71-95-122-38.dhcp.mdfd.or.charter.com) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * bearkitten (~bearkitte@cpe-76-172-86-115.socal.res.rr.com) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * devicenull (sid4013@id-4013.ealing.irccloud.com) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * eternaleye (~eternaley@50.245.141.73) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * KindOne (kindone@0001a7db.user.oftc.net) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * scuttle|afk (~scuttle@nat-pool-rdu-t.redhat.com) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * off_rhoden (~off_rhode@transit-86-181-132-209.redhat.com) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * telnoratti (~telnoratt@pr0n.vtluug.org) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * loicd (~loicd@cmd179.fsffrance.org) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * sage (~quassel@2607:f298:6050:709d:6dc1:2e68:ab33:6549) Quit (resistance.oftc.net graviton.oftc.net)
[11:57] * Karcaw (~evan@71-95-122-38.dhcp.mdfd.or.charter.com) has joined #ceph
[11:57] * jordanP (~jordan@scality-jouf-2-194.fib.nerim.net) has joined #ceph
[11:57] * _robbat2|irssi (nobody@www2.orbis-terrarum.net) has joined #ceph
[11:57] * ivotron (~ivotron@c-67-169-145-20.hsd1.ca.comcast.net) has joined #ceph
[11:57] * `10 (~10@69.169.91.14) has joined #ceph
[11:57] * scuttle|afk (~scuttle@nat-pool-rdu-t.redhat.com) has joined #ceph
[11:57] * hchen (~hchen@nat-pool-bos-t.redhat.com) has joined #ceph
[11:57] * bkopilov (~bkopilov@bzq-109-66-56-13.red.bezeqint.net) has joined #ceph
[11:57] * mattt (~mattt@lnx1.defunct.ca) has joined #ceph
[11:57] * KindOne (kindone@0001a7db.user.oftc.net) has joined #ceph
[11:57] * Hemanth (~Hemanth@121.244.87.117) has joined #ceph
[11:57] * jwilkins (~jowilkin@c-67-180-123-48.hsd1.ca.comcast.net) has joined #ceph
[11:57] * bilco105 (~bilco105@irc.bilco105.com) has joined #ceph
[11:57] * eternaleye (~eternaley@50.245.141.73) has joined #ceph
[11:57] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[11:57] * linjan_ (~linjan@213.80.239.50) has joined #ceph
[11:58] * loicd (~loicd@cmd179.fsffrance.org) has joined #ceph
[11:58] * bearkitten (~bearkitte@cpe-76-172-86-115.socal.res.rr.com) has joined #ceph
[11:58] * jdillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) has joined #ceph
[11:58] * dgurtner (~dgurtner@149.11.161.243) has joined #ceph
[11:58] * c4tech (~c4tech@199.91.185.156) has joined #ceph
[11:58] * championofcyrodi (~championo@50-205-35-98-static.hfc.comcastbusiness.net) has joined #ceph
[11:58] * geli (~geli@geli-2015.its.utas.edu.au) has joined #ceph
[11:59] * karnan (~karnan@106.216.164.93) has joined #ceph
[11:59] * jpieper (~josh@209-6-39-224.c3-0.smr-ubr2.sbo-smr.ma.cable.rcn.com) has joined #ceph
[11:59] * aarontc (~aarontc@2001:470:e893::1:1) has joined #ceph
[11:59] * Pintomatic (sid25118@id-25118.ealing.irccloud.com) has joined #ceph
[11:59] * devicenull (sid4013@id-4013.ealing.irccloud.com) has joined #ceph
[11:59] * sage (~quassel@2607:f298:6050:709d:6dc1:2e68:ab33:6549) has joined #ceph
[11:59] * Kingrat (~shiny@2605:a000:161a:c022:2d0e:daff:3f4a:6f64) has joined #ceph
[11:59] * telnoratti (~telnoratt@pr0n.vtluug.org) has joined #ceph
[12:01] * haomaiwang (~haomaiwan@li817-120.members.linode.com) Quit (Remote host closed the connection)
[12:01] * Jahkeup (Jahkeup@gemini.ca.us.panicbnc.net) has joined #ceph
[12:01] * jklare (~jklare@185.27.181.36) has joined #ceph
[12:02] * haomaiwang (~haomaiwan@li817-120.members.linode.com) has joined #ceph
[12:02] * epf (epf@155.254.32.82) has joined #ceph
[12:02] * off_rhoden (~off_rhode@transit-86-181-132-209.redhat.com) has joined #ceph
[12:05] * Miho (~Xerati@5NZAAGLLD.tor-irc.dnsbl.oftc.net) Quit ()
[12:05] * Pulec1 (~Rosenblut@94.102.49.198) has joined #ceph
[12:07] * rendar (~I@host16-175-dynamic.7-87-r.retail.telecomitalia.it) has joined #ceph
[12:11] * championofcyrodi (~championo@50-205-35-98-static.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[12:11] * derjohn_mobi (~aj@b2b-94-79-172-98.unitymedia.biz) has joined #ceph
[12:12] * kefu is now known as kefu|afk
[12:12] * kefu|afk (~kefu@114.92.110.67) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[12:13] * jclm (~jclm@119.17.40.130) has joined #ceph
[12:13] * Mika_c (~Mika@122.146.93.152) Quit (Quit: Leaving)
[12:14] * elder (~elder@c-24-245-18-91.hsd1.mn.comcast.net) has joined #ceph
[12:14] * ChanServ sets mode +o elder
[12:15] * owasserm (~owasserm@52D9864F.cm-11-1c.dynamic.ziggo.nl) has joined #ceph
[12:15] * arbrandes_ (~arbrandes@189.110.5.98) has joined #ceph
[12:16] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[12:16] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[12:17] * zhaochao (~zhaochao@125.39.8.233) Quit (Quit: ChatZilla 0.9.92 [Iceweasel 38.2.0/20150811213123])
[12:18] <delaf> hello
[12:19] * bilco105 is now known as bilco105_
[12:19] <delaf> does anybody already tried to create a ceph cluster with ATOM D525 CPUs ?
[12:20] <delaf> network will be 1gb/s, each node with 2 OSD + 1SSD for journal
[12:20] <delaf> 1TB OSD sata
[12:21] <bjozet> delaf: I havn't, but please do, and post results! :D
[12:21] <delaf> ^^
[12:21] * bitserker (~toni@47.61.240.151) Quit (Quit: Leaving.)
[12:22] * bitserker (~toni@47.61.240.151) has joined #ceph
[12:22] * bitserker (~toni@47.61.240.151) Quit ()
[12:22] * bitserker (~toni@47.61.240.151) has joined #ceph
[12:22] <delaf> do you know which disk size we should take for journal if my OSD is 1TB ?
[12:23] * bitserker (~toni@47.61.240.151) Quit ()
[12:23] * bitserker (~toni@47.61.240.151) has joined #ceph
[12:23] * bilco105_ is now known as bilco105
[12:29] * jclm (~jclm@119.17.40.130) Quit (Ping timeout: 480 seconds)
[12:35] * Pulec1 (~Rosenblut@1ADAAARRD.tor-irc.dnsbl.oftc.net) Quit ()
[12:35] * eXeler0n (~aldiyen@9S0AADGWB.tor-irc.dnsbl.oftc.net) has joined #ceph
[12:37] * rdas (~rdas@121.244.87.116) Quit (Quit: Leaving)
[12:40] * Satiel (~giordano@185.3.39.54) has joined #ceph
[12:41] * bitserker (~toni@47.61.240.151) Quit (Read error: Connection reset by peer)
[12:42] * bitserker (~toni@47.61.240.151) has joined #ceph
[12:42] * bitserker (~toni@47.61.240.151) Quit ()
[12:42] * bitserker (~toni@47.61.240.151) has joined #ceph
[12:43] * kawa2014 (~kawa@89.184.114.246) Quit (Ping timeout: 480 seconds)
[12:50] * smerz (~ircircirc@37.74.194.90) has joined #ceph
[12:53] * rotbeard (~redbeard@cm-171-100-223-199.revip10.asianet.co.th) Quit (Quit: Leaving)
[12:54] * kawa2014 (~kawa@89.184.114.246) has joined #ceph
[12:55] * shyu (~Shanzhi@119.254.120.66) Quit (Remote host closed the connection)
[12:56] * bilco105 is now known as bilco105_
[12:57] * rotbeard (~redbeard@cm-171-100-223-199.revip10.asianet.co.th) has joined #ceph
[12:59] * brutuscat (~brutuscat@234.Red-79-151-98.dynamicIP.rima-tde.net) Quit (Remote host closed the connection)
[13:01] * haomaiwang (~haomaiwan@li817-120.members.linode.com) Quit (Remote host closed the connection)
[13:01] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[13:01] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[13:02] * haomaiwang (~haomaiwan@li817-120.members.linode.com) has joined #ceph
[13:05] * kefu (~kefu@114.92.110.67) has joined #ceph
[13:05] * eXeler0n (~aldiyen@9S0AADGWB.tor-irc.dnsbl.oftc.net) Quit ()
[13:05] * darks1 (~Tralin|Sl@91.213.8.235) has joined #ceph
[13:06] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[13:06] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[13:06] <samw> I suspect these to be the cause:
[13:07] <samw> rados ls -p .be-east.rgw.buckets | grep sanity
[13:07] <samw> be-east.5436.1__:2bpm.1OR-cqyOLUHek8m2RdPVRZ.pDT__sanity be-east.5436.1__sanity
[13:07] <samw> be-east.5436.1__:2vBijaGnVQF4Q0IjZPeyZSKeUmBGn9X__sanity be-east.5436.1__sanity
[13:07] <samw> be-east.5436.1__:4JTCVFxB1qoDWPu1nhuMDuZ3QNPaq5n__sanity be-east.5436.1__sanity
[13:07] <samw> be-east.5436.1__:9jFwd8xvqJMdrqZuM8Au4mi9M62ikyo__sanity be-east.5436.1__sanity
[13:07] <samw> be-east.5436.1__:BlfbGYGvLi92QPSiabT2mP7OeuETz0P__sanity be-east.5436.1__sanity
[13:07] <samw> be-east.5436.1__:MigpcpJKkan7Po6vBsQsSD.hEIRWuim__sanity be-east.5436.1__sanity
[13:07] <samw> be-east.5436.1__:QDTxD5p0AmVlPW4v8OPU3vtDLzenj4y__sanity be-east.5436.1__sanity
[13:07] <samw> be-east.5436.1__:S43EiNAk5hOkzgfbOynbOZOuLtUv0SB__sanity be-east.5436.1__sanity
[13:07] <samw> be-east.5436.1__:UKlOVMQBQnlK20BHJPyvnG6m.2ogBRW__sanity be-east.5436.1__sanity
[13:07] <samw> be-east.5436.1__:kkb6muzJgREie6XftdEJdFHxR2MaFeB__sanity be-east.5436.1__sanity
[13:07] <samw> be-east.5436.1__:oqPhWzFDSQ-sNPtppsl1tPjoryaHNZY__sanity be-east.5436.1__sanity
[13:07] <samw> be-east.5436.1__:pLhygPGKf3uw7C7OxSJNCw8rQEMOw5l__sanity be-east.5436.1__sanity
[13:07] <samw> be-east.5436.1__:tO1Nf3S2WOfmcnKVPv0tMeXbwa5JR36__sanity be-east.5436.1__sanity
[13:07] <samw> be-east.5436.1__:ye4oRwDDh1cGckbMbIo56nQvM7OEyPM__sanity be-east.5436.1__sanity
[13:07] <samw> be-east.5436.1___sanity be-east.5436.1__sanity
[13:07] <samw> would it be save and/or help to remove those with "rados rm"?
[13:07] <samw> and try a "bucket check --fix --check-objects"?
[13:12] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[13:12] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[13:16] <samw> ok, completely lost now
[13:16] <samw> rados -p .be-east.rgw.buckets rm "be-east.5436.1__:2bpm.1OR-cqyOLUHek8m2RdPVRZ.pDT__sanity" gives me
[13:16] <samw> error removing .be-east.rgw.buckets>be-east.5436.1__:2bpm.1OR-cqyOLUHek8m2RdPVRZ.pDT__sanity: (2) No such file or directory
[13:19] * ira (~ira@c-71-233-225-22.hsd1.ma.comcast.net) has joined #ceph
[13:20] * dopesong (~dopesong@lb1.mailer.data.lt) Quit (Remote host closed the connection)
[13:32] * kefu (~kefu@114.92.110.67) Quit (Max SendQ exceeded)
[13:33] * kefu (~kefu@114.92.110.67) has joined #ceph
[13:35] * darks1 (~Tralin|Sl@1ADAAARTV.tor-irc.dnsbl.oftc.net) Quit ()
[13:35] * Jourei (~smf68@79.98.107.90) has joined #ceph
[13:36] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit (Ping timeout: 480 seconds)
[13:42] * haomaiwang (~haomaiwan@li817-120.members.linode.com) Quit (Remote host closed the connection)
[13:48] * dopesong (~dopesong@lb1.mailer.data.lt) has joined #ceph
[13:55] * yghannam (~yghannam@0001f8aa.user.oftc.net) has joined #ceph
[13:57] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[13:57] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[13:58] * Hemanth (~Hemanth@121.244.87.117) Quit (Ping timeout: 480 seconds)
[14:00] * dgurtner (~dgurtner@149.11.161.243) Quit (Ping timeout: 480 seconds)
[14:05] * bilco105_ is now known as bilco105
[14:05] * Jourei (~smf68@1ADAAARVA.tor-irc.dnsbl.oftc.net) Quit ()
[14:06] * kefu (~kefu@114.92.110.67) Quit (Max SendQ exceeded)
[14:06] * Shesh (~Esge@edwardsnowden0.torservers.net) has joined #ceph
[14:07] * kefu (~kefu@114.92.110.67) has joined #ceph
[14:08] * patate (~martin@208.84.104.158) has joined #ceph
[14:10] * T1w (~jens@node3.survey-it.dk) Quit (Ping timeout: 480 seconds)
[14:10] * kefu is now known as kefu|afk
[14:11] * kefu|afk is now known as kefu
[14:21] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[14:21] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[14:30] * dgurtner (~dgurtner@149.11.161.243) has joined #ceph
[14:35] * nardial (~ls@dslb-178-006-188-098.178.006.pools.vodafone-ip.de) has joined #ceph
[14:35] * Shesh (~Esge@1ADAAARWH.tor-irc.dnsbl.oftc.net) Quit ()
[14:41] * fmenegoni (~fmenegoni@89-97-231-24.ip19.fastwebnet.it) has joined #ceph
[14:44] * i_m (~ivan.miro@deibp9eh1--blueice1n4.emea.ibm.com) has joined #ceph
[14:46] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[14:46] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Remote host closed the connection)
[14:46] * karnan (~karnan@106.216.164.93) Quit (Ping timeout: 480 seconds)
[14:50] * mhack (~mhack@nat-pool-bos-t.redhat.com) has joined #ceph
[14:51] * wschulze (~wschulze@cpe-69-206-242-167.nyc.res.rr.com) has joined #ceph
[14:51] * yanzheng (~zhyan@182.139.20.109) Quit (Quit: This computer has gone to sleep)
[14:51] * pdrakeweb (~pdrakeweb@104.247.39.34) has joined #ceph
[14:52] * yanzheng (~zhyan@182.139.20.109) has joined #ceph
[14:55] * georgem (~Adium@65-110-211-254.cpe.pppoe.ca) has joined #ceph
[14:59] * karnan (~karnan@106.51.232.229) has joined #ceph
[15:02] * georgem (~Adium@65-110-211-254.cpe.pppoe.ca) Quit (Quit: Leaving.)
[15:02] * fmenegoni (~fmenegoni@89-97-231-24.ip19.fastwebnet.it) Quit (Quit: Leaving)
[15:02] * karnan (~karnan@106.51.232.229) Quit ()
[15:06] * Drezil (~n0x1d@104.255.64.26) has joined #ceph
[15:10] * edrian (~edrian@globe.ph.syrahost.com) Quit (Quit: WeeChat 0.4.2)
[15:16] * shaunm (~shaunm@50-5-225-172.dynamic.fuse.net) has joined #ceph
[15:17] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[15:17] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[15:18] * analbeard (~shw@support.memset.com) Quit (Ping timeout: 480 seconds)
[15:20] * brad_mssw (~brad@66.129.88.50) has joined #ceph
[15:21] * marrusl (~mark@cpe-67-247-9-253.nyc.res.rr.com) has joined #ceph
[15:22] * rdas (~rdas@121.244.87.116) has joined #ceph
[15:23] * amote (~amote@121.244.87.116) Quit (Quit: Leaving)
[15:23] * xdeller (~xdeller@h195-91-128-218.ln.rinet.ru) has joined #ceph
[15:31] * championofcyrodi (~championo@50-205-35-98-static.hfc.comcastbusiness.net) has joined #ceph
[15:31] * rdas (~rdas@121.244.87.116) Quit (Quit: Leaving)
[15:31] * analbeard (~shw@5.153.255.226) has joined #ceph
[15:33] * kmARC (~kmARC@2001:620:20:16:31b3:f949:28d2:88d3) has joined #ceph
[15:34] * kmARC (~kmARC@2001:620:20:16:31b3:f949:28d2:88d3) Quit ()
[15:35] * Rickus (~Rickus@office.protected.ca) Quit (Read error: Connection reset by peer)
[15:35] * Drezil (~n0x1d@1ADAAARZM.tor-irc.dnsbl.oftc.net) Quit ()
[15:36] * Ian2128 (~Throlkim@r3.geoca.st) has joined #ceph
[15:41] * nisha (~nisha@14.139.251.107) has joined #ceph
[15:43] * harold (~hamiller@71-94-227-123.dhcp.mdfd.or.charter.com) has joined #ceph
[15:43] * harold (~hamiller@71-94-227-123.dhcp.mdfd.or.charter.com) Quit ()
[15:44] * dgurtner (~dgurtner@149.11.161.243) Quit (Ping timeout: 480 seconds)
[15:45] * smerz (~ircircirc@37.74.194.90) Quit (Quit: Leaving)
[15:51] * jrankin (~jrankin@d53-64-170-236.nap.wideopenwest.com) has joined #ceph
[15:52] * fsimonce (~simon@host93-234-dynamic.252-95-r.retail.telecomitalia.it) has joined #ceph
[15:52] * dgurtner (~dgurtner@149.11.161.243) has joined #ceph
[15:58] * pdrakeweb (~pdrakeweb@104.247.39.34) Quit (Remote host closed the connection)
[16:00] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[16:00] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[16:00] * brad_mssw (~brad@66.129.88.50) Quit (Remote host closed the connection)
[16:02] * kanagaraj (~kanagaraj@101.222.235.38) has joined #ceph
[16:02] * brutuscat (~brutuscat@234.Red-79-151-98.dynamicIP.rima-tde.net) has joined #ceph
[16:03] * kefu is now known as kefu|afk
[16:03] * dyasny (~dyasny@173.231.115.59) has joined #ceph
[16:04] * pdrakeweb (~pdrakeweb@104.247.39.34) has joined #ceph
[16:04] * brad_mssw (~brad@66.129.88.50) has joined #ceph
[16:04] * kanagaraj (~kanagaraj@101.222.235.38) Quit ()
[16:05] * Ian2128 (~Throlkim@5NZAAGLUE.tor-irc.dnsbl.oftc.net) Quit ()
[16:06] * Jourei1 (~chrisinaj@luxemburg.gtor.org) has joined #ceph
[16:06] * lcurtis (~lcurtis@47.19.105.250) has joined #ceph
[16:07] * maartendq (~maarten@94-224-160-55.access.telenet.be) has joined #ceph
[16:07] * b0e (~aledermue@213.95.25.82) Quit (Quit: Leaving.)
[16:09] * nisha (~nisha@14.139.251.107) Quit (Ping timeout: 480 seconds)
[16:13] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[16:13] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[16:18] * cholcombe (~chris@c-73-180-29-35.hsd1.or.comcast.net) has joined #ceph
[16:18] * nisha (~nisha@14.139.251.107) has joined #ceph
[16:22] * nardial (~ls@dslb-178-006-188-098.178.006.pools.vodafone-ip.de) Quit (Quit: Leaving)
[16:23] * kilinrax (~alex@94.228.69.146) has joined #ceph
[16:24] <kilinrax> Hello. I'm using librados and hitting ord_max_write_size. What's the recommended way to write files as objects when the files are over that 90Mb size?
[16:24] * bilco105 is now known as bilco105_
[16:26] * sankarshan (~sankarsha@121.244.87.117) Quit (Quit: Are you sure you want to quit this channel (Cancel/Ok) ?)
[16:26] * haomaiwang (~haomaiwan@li817-120.members.linode.com) has joined #ceph
[16:27] * pdrakeweb (~pdrakeweb@104.247.39.34) Quit (Remote host closed the connection)
[16:27] * kefu|afk (~kefu@114.92.110.67) Quit (Max SendQ exceeded)
[16:28] * kefu (~kefu@114.92.110.67) has joined #ceph
[16:30] * pdrakeweb (~pdrakeweb@104.247.39.34) has joined #ceph
[16:32] * bilco105_ is now known as bilco105
[16:35] * ircolle (~Adium@2601:285:201:2bf9:b0df:2f64:26ba:2e0a) has joined #ceph
[16:35] * reed (~reed@75-101-54-18.dsl.static.fusionbroadband.com) has joined #ceph
[16:36] * Jourei1 (~chrisinaj@62SAAAU5K.tor-irc.dnsbl.oftc.net) Quit ()
[16:37] * sese_ (~Pieman@tor1.digineo.de) has joined #ceph
[16:43] * reset11 (~reset11@141.244.133.89) Quit (Ping timeout: 480 seconds)
[16:50] <gregsfortytwo> kilinrax: depends what you're trying to do
[16:50] <gregsfortytwo> if your objects are variably-sized you might want to do some striping (like RBD, RGW, and CephFS do)
[16:51] <gregsfortytwo> if you just want them to be single objects you can write them in multiple operations, each of which is individually less than 90 MB
[16:52] <gregsfortytwo> but keep in mind that objects of that size are not well-tested in RADOS, you've got a higher probability of hitting bugs than people generally do if you take that route
[16:52] <kilinrax> Most of my files/objects are 1-2Mb, but some are just over the threshold. 123Mb is the largest, currently, though that will rise with time.
[16:54] * scuttle|afk is now known as scuttlemonkey
[16:54] <kilinrax> I'm putting them in ceph partically to exnsure they're safe/replicated/et c. and partically so other applications can retrieve them, only needing a small payload of metadata (ceph oid).
[16:55] <kilinrax> Are there any solutions for striping that wouldn't require all applications to know how the files are segmented/import the striping library?
[16:55] * analbeard (~shw@5.153.255.226) Quit (Quit: Leaving.)
[16:59] <gregsfortytwo> well, you'd obviously need the striping code, but you are already getting librados in somehow so presumably it could go in the same way?
[17:01] * haomaiwang (~haomaiwan@li817-120.members.linode.com) Quit (Remote host closed the connection)
[17:02] * haomaiwang (~haomaiwan@li817-120.members.linode.com) has joined #ceph
[17:02] * yanzheng (~zhyan@182.139.20.109) Quit (Quit: This computer has gone to sleep)
[17:02] <Peltzi> there is already implementation for that called librados-striper
[17:02] <Peltzi> or something like that
[17:03] <Peltzi> I remember reading about it not long ago
[17:03] <kilinrax> Oh, is there? That could be very helpful, thanks Peltzi.
[17:03] <Peltzi> https://github.com/ceph/ceph/blob/master/src/libradosstriper/libradosstriper.cc
[17:03] <Peltzi> no sure were any sort of documentatios for that it :)
[17:03] <Peltzi> but 0.83 release notes tell that it was generic libradosstriper API was released :)
[17:03] <Peltzi> *is
[17:04] * pdrakeweb (~pdrakeweb@104.247.39.34) Quit (Remote host closed the connection)
[17:06] * sese_ (~Pieman@62SAAAU7A.tor-irc.dnsbl.oftc.net) Quit ()
[17:06] * Linkshot (~mLegion@tor-exit.kartofelzik.info) has joined #ceph
[17:07] * dgurtner (~dgurtner@149.11.161.243) Quit (Ping timeout: 480 seconds)
[17:07] * Concubidated (~Adium@71.21.5.251) has joined #ceph
[17:10] * zack_dolby (~textual@pa3b3a1.tokynt01.ap.so-net.ne.jp) has joined #ceph
[17:10] * kefu is now known as kefu|afk
[17:10] * kefu|afk is now known as kefu
[17:15] * TMM (~hp@sams-office-nat.tomtomgroup.com) Quit (Quit: Ex-Chat)
[17:17] * jrankin (~jrankin@d53-64-170-236.nap.wideopenwest.com) Quit (Quit: Leaving)
[17:18] <samw> anyone any idea if this tracker also fixes issues with versioned objects? http://tracker.ceph.com/issues/12242
[17:19] * fdmanana (~fdmanana@2001:8a0:6dfd:6d01:a859:709b:67fc:b0e2) Quit (Quit: Leaving)
[17:20] * bilco105 is now known as bilco105_
[17:27] * snakamoto (~Adium@cpe-76-91-202-90.socal.res.rr.com) has joined #ceph
[17:27] * bilco105_ is now known as bilco105
[17:36] * Linkshot (~mLegion@5NZAAGLX2.tor-irc.dnsbl.oftc.net) Quit ()
[17:36] * VampiricPadraig (~Inverness@176.10.99.209) has joined #ceph
[17:36] * yanzheng (~zhyan@182.139.20.109) has joined #ceph
[17:40] * bitserker (~toni@47.61.240.151) Quit (Quit: Leaving.)
[17:40] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[17:40] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[17:43] * reset11 (~reset11@chello080108240025.2.14.vie.surfer.at) has joined #ceph
[17:44] * fdmanana (~fdmanana@2001:8a0:6dfd:6d01:2ab2:bdff:fe87:46f) has joined #ceph
[17:44] * pdrakeweb (~pdrakeweb@104.247.39.34) has joined #ceph
[17:45] * pdrakeweb (~pdrakeweb@104.247.39.34) Quit (Remote host closed the connection)
[17:47] * snakamoto (~Adium@cpe-76-91-202-90.socal.res.rr.com) Quit (Quit: Leaving.)
[17:47] * jordanP (~jordan@scality-jouf-2-194.fib.nerim.net) Quit (Quit: Leaving)
[17:48] * gregmark (~Adium@68.87.42.115) has joined #ceph
[17:49] * fsimonce (~simon@host93-234-dynamic.252-95-r.retail.telecomitalia.it) Quit (Quit: Coyote finally caught me)
[17:49] * snakamoto (~Adium@cpe-76-91-202-90.socal.res.rr.com) has joined #ceph
[17:50] * yanzheng (~zhyan@182.139.20.109) Quit (Quit: This computer has gone to sleep)
[17:55] * derjohn_mobi (~aj@b2b-94-79-172-98.unitymedia.biz) Quit (Ping timeout: 480 seconds)
[17:57] * snakamoto (~Adium@cpe-76-91-202-90.socal.res.rr.com) Quit (Ping timeout: 480 seconds)
[18:01] * haomaiwang (~haomaiwan@li817-120.members.linode.com) Quit (Remote host closed the connection)
[18:02] * haomaiwang (~haomaiwan@li817-120.members.linode.com) has joined #ceph
[18:05] * moore (~moore@64.202.160.88) has joined #ceph
[18:06] * VampiricPadraig (~Inverness@62SAAAU91.tor-irc.dnsbl.oftc.net) Quit ()
[18:06] * lmg (~Curt`@192.42.115.101) has joined #ceph
[18:10] * reed (~reed@75-101-54-18.dsl.static.fusionbroadband.com) Quit (Quit: Ex-Chat)
[18:11] * branto (~branto@178-253-131-8.3pp.slovanet.sk) Quit (Quit: Leaving.)
[18:11] * reed (~reed@75-101-54-18.dsl.static.fusionbroadband.com) has joined #ceph
[18:12] * rotbeard (~redbeard@cm-171-100-223-199.revip10.asianet.co.th) Quit (Ping timeout: 480 seconds)
[18:13] <olid1989> my writing performance on the ceph cluster is totally rubbish after i switched
[18:13] <olid1989> ceph osd pool set rbd size
[18:13] <olid1989> from 2 to 3
[18:13] <olid1989> changing it back to 2 does not help
[18:13] <olid1989> any idea how to investigate this ?
[18:16] * nardial (~ls@dslb-178-006-188-098.178.006.pools.vodafone-ip.de) has joined #ceph
[18:17] * yguang11 (~yguang11@2001:4998:effd:600:6015:46c6:3ad7:287a) has joined #ceph
[18:20] * ngoswami_ (~ngoswami@121.244.87.116) Quit (Quit: Leaving)
[18:29] * i_m (~ivan.miro@deibp9eh1--blueice1n4.emea.ibm.com) Quit (Read error: Connection reset by peer)
[18:30] * Wes_d (~wes_d@140.247.242.44) has joined #ceph
[18:31] * nardial (~ls@dslb-178-006-188-098.178.006.pools.vodafone-ip.de) Quit (Quit: Leaving)
[18:32] <Wes_d> Hi all, still getting my feet wet. I have built a cluster with deph deploy. I have 24 osds (all up and in) I have 1024 pgs on my pools. 272 of my pgs are degraded. I found that degraded means that objects have not been replicated to the correct replication level which is set at 3, however, i have put no objects in my cluster yet, so wondering how that could be???. Thanks!
[18:35] <Wes_d> Also, wondering how degraded is different than undersized, as the description makes them sound identical http://ceph.com/docs/master/rados/operations/pg-states/
[18:36] * lmg (~Curt`@5NZAAGL0B.tor-irc.dnsbl.oftc.net) Quit ()
[18:36] * measter (~rikai@217.12.204.104) has joined #ceph
[18:36] * kawa2014 (~kawa@89.184.114.246) Quit (Quit: Leaving)
[18:47] * jasuarez (~jasuarez@78.Red-83-37-130.dynamicIP.rima-tde.net) Quit (Quit: WeeChat 1.2)
[18:49] * samw (~sam@94-224-154-150.access.telenet.be) Quit (Quit: Leaving)
[18:56] * Aeso_ (~AesoSpade@c-68-37-97-11.hsd1.mi.comcast.net) has joined #ceph
[18:57] * mykola (~Mikolaj@91.225.200.81) has joined #ceph
[19:00] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[19:00] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[19:01] * haomaiwang (~haomaiwan@li817-120.members.linode.com) Quit (Remote host closed the connection)
[19:01] * Aeso (~AesoSpade@mail.sureauction.net) Quit (Ping timeout: 480 seconds)
[19:02] * haomaiwang (~haomaiwan@li817-120.members.linode.com) has joined #ceph
[19:03] * Aeso_ is now known as Aeso
[19:06] * measter (~rikai@5NZAAGL1K.tor-irc.dnsbl.oftc.net) Quit ()
[19:06] * RaidSoft (~KeeperOfT@Relay-J.tor-exit.network) has joined #ceph
[19:09] * kefu is now known as kefu|afk
[19:10] * linjan_ (~linjan@213.80.239.50) Quit (Ping timeout: 480 seconds)
[19:13] * snakamoto (~Adium@192.16.26.2) has joined #ceph
[19:15] * kefu|afk (~kefu@114.92.110.67) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[19:16] * danieagle (~Daniel@187.10.31.191) has joined #ceph
[19:24] * brutuscat (~brutuscat@234.Red-79-151-98.dynamicIP.rima-tde.net) Quit (Remote host closed the connection)
[19:26] * jbautista- (~wushudoin@transit-86-181-132-209.redhat.com) has joined #ceph
[19:34] * wushudoin_ (~wushudoin@38.140.108.2) Quit (Ping timeout: 480 seconds)
[19:35] * jbautista- (~wushudoin@transit-86-181-132-209.redhat.com) Quit (Ping timeout: 480 seconds)
[19:35] * jbautista- (~wushudoin@38.140.108.2) has joined #ceph
[19:36] * RaidSoft (~KeeperOfT@1ADAAASDQ.tor-irc.dnsbl.oftc.net) Quit ()
[19:36] * Heliwr (~ghostnote@torland1-this.is.a.tor.exit.server.torland.is) has joined #ceph
[19:41] * bloatyfloat_ (~bloatyflo@46.37.172.252.srvlist.ukfast.net) has joined #ceph
[19:44] * doppelgrau (~doppelgra@pd956d116.dip0.t-ipconnect.de) has joined #ceph
[19:44] * bloatyfloat__ (~bloatyflo@46.37.172.253.srvlist.ukfast.net) has joined #ceph
[19:45] * linjan_ (~linjan@79.126.79.215) has joined #ceph
[19:47] * bloatyfloat (~bloatyflo@46.37.172.253.srvlist.ukfast.net) Quit (Ping timeout: 480 seconds)
[19:48] * Kupo1 (~tyler.wil@23.111.254.159) has joined #ceph
[19:48] * bloatyfloat__ is now known as bloatyfloat
[19:49] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[19:50] * bloatyfloat_ (~bloatyflo@46.37.172.252.srvlist.ukfast.net) Quit (Ping timeout: 480 seconds)
[19:53] * Satiel (~giordano@185.3.39.54) Quit (Quit: Satiel)
[19:58] * lovejoy (~lovejoy@cpc69386-oxfd28-2-0-cust928.4-3.cable.virginm.net) has joined #ceph
[19:58] * lovejoy (~lovejoy@cpc69386-oxfd28-2-0-cust928.4-3.cable.virginm.net) Quit ()
[20:01] * haomaiwang (~haomaiwan@li817-120.members.linode.com) Quit (Remote host closed the connection)
[20:02] * haomaiwang (~haomaiwan@li817-120.members.linode.com) has joined #ceph
[20:02] * daviddcc (~dcasier@77.151.197.84) has joined #ceph
[20:06] * Heliwr (~ghostnote@5NZAAGL3N.tor-irc.dnsbl.oftc.net) Quit ()
[20:16] * nisha (~nisha@14.139.251.107) Quit (Ping timeout: 480 seconds)
[20:20] * xarses_ (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) Quit (Remote host closed the connection)
[20:20] * xarses_ (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) has joined #ceph
[20:20] * fdmanana (~fdmanana@2001:8a0:6dfd:6d01:2ab2:bdff:fe87:46f) Quit (Quit: Leaving)
[20:28] <olid1989> ceph osd perf
[20:28] <olid1989> osd fs_commit_latency(ms) fs_apply_latency(ms)
[20:28] <olid1989> 0 1571 1653
[20:28] <olid1989> 1 1788 1898
[20:28] <olid1989> 2 1705 1795
[20:28] <olid1989> 3 33 39
[20:28] <olid1989> 4 33 41
[20:28] <olid1989> 5 34 39
[20:28] <olid1989> 6 322 331
[20:28] <olid1989> 7 322 329
[20:28] <olid1989> 8 346 357
[20:28] <olid1989> 9 20 23
[20:28] <olid1989> 10 21 28
[20:29] <olid1989> 11 21 26
[20:29] <olid1989> that does not look good i think, maybe thats why the write performance of the cluster is with 20/30 MB/s quiet shitty
[20:29] <olid1989> how to fix that ? ^^;
[20:29] * xarses_ (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[20:35] * nisha (~nisha@14.139.251.107) has joined #ceph
[20:36] * blip2 (~AG_Clinto@r3.geoca.st) has joined #ceph
[20:38] <Aeso> looks like osd 0, 1, and 2 are very slow. Are they all on the same host?
[20:38] <Aeso> re: olid1989
[20:38] <olid1989> well the strange thing is, that nothing changed, before this cluster had 130 MB/s ( with gbit fully ok )
[20:39] <olid1989> then i changed:
[20:39] <olid1989> ceph osd pool set rbd size
[20:39] <olid1989> from 2 to 3
[20:39] <olid1989> he did the copy and sync
[20:40] <olid1989> and after that, on this two nodes ( osd 0,1,2 = hostA, osd 6,7,8 = hostC )
[20:40] <olid1989> gone rubbish
[20:40] <olid1989> right now i am removing the osd's and recreate them
[20:40] <olid1989> and check if its still there
[20:46] * xarses_ (~xarses@12.164.168.117) has joined #ceph
[20:48] <rkeene> So I figured out why Ceph was reporting my systems were out of sync
[20:48] <rkeene> (time sync)
[20:50] * Destreyf (~quassel@email.newagecomputers.info) has joined #ceph
[20:52] <rkeene> It (from my observation, not code diving) uses a rolling average to determine the time sync, so if Ceph is started while the times are way out of sync then they jump into sync, Ceph will take a long time (sampling every 5 minutes) to get that average down to 0.05s
[20:52] * c4tech (~c4tech@199.91.185.156) Quit (Read error: Connection reset by peer)
[20:53] * c4tech (~c4tech@199.91.185.156) has joined #ceph
[21:01] * haomaiwang (~haomaiwan@li817-120.members.linode.com) Quit (Remote host closed the connection)
[21:01] * haomaiwang (~haomaiwan@li817-120.members.linode.com) has joined #ceph
[21:06] * blip2 (~AG_Clinto@62SAAAVJN.tor-irc.dnsbl.oftc.net) Quit ()
[21:06] * Chaos_Llama (~skrblr@tor.ssessess.es) has joined #ceph
[21:12] * moore (~moore@64.202.160.88) Quit (Remote host closed the connection)
[21:15] * xarses (~xarses@12.164.168.117) has joined #ceph
[21:31] * visbits (~textual@8.29.138.28) has joined #ceph
[21:36] * Chaos_Llama (~skrblr@1ADAAASKI.tor-irc.dnsbl.oftc.net) Quit ()
[21:36] * ChauffeR1 (~redbeast1@atlantic480.us.unmetered.com) has joined #ceph
[21:38] * moore (~moore@64.202.160.233) has joined #ceph
[21:50] * bkopilov (~bkopilov@bzq-109-66-56-13.red.bezeqint.net) Quit (Ping timeout: 480 seconds)
[21:51] * bkopilov (~bkopilov@bzq-109-66-149-27.red.bezeqint.net) has joined #ceph
[21:51] * shaunm (~shaunm@50-5-225-172.dynamic.fuse.net) Quit (Ping timeout: 480 seconds)
[21:55] * brad_mssw (~brad@66.129.88.50) Quit (Quit: Leaving)
[21:58] * ivotron (~ivotron@c-67-169-145-20.hsd1.ca.comcast.net) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[22:01] * haomaiwang (~haomaiwan@li817-120.members.linode.com) Quit (Remote host closed the connection)
[22:02] * haomaiwang (~haomaiwan@li817-120.members.linode.com) has joined #ceph
[22:04] * wkennington_ is now known as wkennington
[22:05] * maartendq (~maarten@94-224-160-55.access.telenet.be) Quit (Quit: Leaving)
[22:06] * ChauffeR1 (~redbeast1@1ADAAASLP.tor-irc.dnsbl.oftc.net) Quit ()
[22:06] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[22:06] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[22:14] * dyasny (~dyasny@173.231.115.59) Quit (Ping timeout: 480 seconds)
[22:16] * dyasny (~dyasny@173.231.115.59) has joined #ceph
[22:17] * linjan_ (~linjan@79.126.79.215) Quit (Ping timeout: 480 seconds)
[22:18] * arbrandes_ (~arbrandes@189.110.5.98) Quit (Quit: Leaving)
[22:30] * Aeso (~AesoSpade@c-68-37-97-11.hsd1.mi.comcast.net) Quit (Quit: Leaving)
[22:31] * bkopilov (~bkopilov@bzq-109-66-149-27.red.bezeqint.net) Quit (Ping timeout: 480 seconds)
[22:40] * bkopilov (~bkopilov@bzq-79-180-137-29.red.bezeqint.net) has joined #ceph
[22:45] * nisha (~nisha@14.139.251.107) Quit (Quit: Leaving)
[22:47] * dyasny (~dyasny@173.231.115.59) Quit (Ping timeout: 480 seconds)
[22:50] * rendar (~I@host16-175-dynamic.7-87-r.retail.telecomitalia.it) Quit (Ping timeout: 480 seconds)
[22:51] * mykola (~Mikolaj@91.225.200.81) Quit (Quit: away)
[22:52] * rendar (~I@host16-175-dynamic.7-87-r.retail.telecomitalia.it) has joined #ceph
[22:59] * xarses (~xarses@12.164.168.117) Quit (Quit: Leaving)
[23:00] * Wes_d (~wes_d@140.247.242.44) Quit (Ping timeout: 480 seconds)
[23:01] * haomaiwang (~haomaiwan@li817-120.members.linode.com) Quit (Remote host closed the connection)
[23:01] * haomaiwang (~haomaiwan@li817-120.members.linode.com) has joined #ceph
[23:06] * Lite (~Shadow386@lumumba.torservers.net) has joined #ceph
[23:06] * yghannam (~yghannam@0001f8aa.user.oftc.net) Quit (Ping timeout: 480 seconds)
[23:10] * mhack (~mhack@nat-pool-bos-t.redhat.com) Quit (Quit: Leaving)
[23:13] * moore (~moore@64.202.160.233) Quit (Remote host closed the connection)
[23:15] * scuttlemonkey is now known as scuttle|afk
[23:20] * owasserm (~owasserm@52D9864F.cm-11-1c.dynamic.ziggo.nl) Quit (Quit: Ex-Chat)
[23:23] * wschulze (~wschulze@cpe-69-206-242-167.nyc.res.rr.com) Quit (Quit: Leaving.)
[23:24] * dgurtner (~dgurtner@178.197.229.78) has joined #ceph
[23:31] <cholcombe> ceph: after the ip:port what does the /number mean in the logs?
[23:31] <cholcombe> is that the pid?
[23:31] * moore (~moore@71-211-73-118.phnx.qwest.net) has joined #ceph
[23:32] * Vince (~Vince@207.66.146.2) has joined #ceph
[23:33] * linuxkidd (~linuxkidd@41.sub-70-195-74.myvzw.com) Quit (Remote host closed the connection)
[23:34] <Vince> Hey guys, what's the best way to get InkTank professional services on site to do a consult/first deployment? I've tried the Inktank site but it's now redirecting to Red Hat's generic services. We're looking to convert 10PB of hardware from another object store to Ceph and need to do a PoC. We're a M&E company based in LA.
[23:35] <ircolle> that's because Inktank was acquired by Red Hat last year :-)
[23:36] * Lite (~Shadow386@9S0AADHA6.tor-irc.dnsbl.oftc.net) Quit ()
[23:36] * tuhnis (~Spessu@aurora.enn.lu) has joined #ceph
[23:36] <Vince> I'm aware - but I don't think the site started redirecting until very recently.
[23:42] * Sysadmin88 (~IceChat77@2.125.96.238) has joined #ceph
[23:45] * marrusl (~mark@cpe-67-247-9-253.nyc.res.rr.com) Quit (Quit: bye!)
[23:48] * dneary (~dneary@nat-pool-bos-u.redhat.com) Quit (Ping timeout: 480 seconds)
[23:48] * dneary_ (~dneary@nat-pool-bos-u.redhat.com) Quit (Ping timeout: 480 seconds)
[23:50] * zaitcev (~zaitcev@2001:558:6001:10:61d7:f51f:def8:4b0f) has joined #ceph
[23:56] * moore (~moore@71-211-73-118.phnx.qwest.net) Quit (Remote host closed the connection)
[23:56] * moore (~moore@64.202.160.233) has joined #ceph

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.