#ceph IRC Log

Index

IRC Log for 2015-11-04

Timestamps are in GMT/BST.

[0:00] * davidzlap1 (~Adium@2605:e000:1313:8003:e83b:414f:4015:1417) Quit (Read error: Connection reset by peer)
[0:00] <mfa298_> unfortunately we bought lots of the smr drives when we were looking at something before ceph, and now we're trying to make them work with ceph. 6TB greens (whilst potentially still not ideal) are much faster.
[0:01] <cetex> yeah. problem is we only have room for 450 hdd's at most with the current hardware, and i'd like to be able to grow so i don't want to fill all slots at once. there's a balance somewhere.
[0:02] * MentalRay (~MentalRay@MTRLPQ42-1176054809.sdsl.bell.ca) Quit (Ping timeout: 480 seconds)
[0:04] <cetex> hm.. it's probably much less than 450 actually. forgot that 40 hosts only have 2.5" slots so 330 drives is more likely in total.
[0:04] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[0:04] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[0:05] <cetex> oh well, bedtime. back tomorrow.
[0:08] <Thunderbird> mfa298_, in your SMR setup do you have an SSD based journal?
[0:10] <mfa298_> not currently, although it's something I've started testing. We had been planning to run a cache tier instead
[0:10] * Icey (~IceyEC_@0001bbad.user.oftc.net) has joined #ceph
[0:10] <Thunderbird> with an SSD journal I would expect performance to get okay again since the direct writes to the journal are the main performance bottleneck
[0:12] <mfa298_> the quick test I did last week did seem to improve things, although that test cluster was still slower than an equivalent test cluster using 6TB WD Greens and no SSD journals
[0:13] <Thunderbird> for my use case we are considering SMR, performance is less of an issue, so was just curious
[0:15] <mfa298_> we're in a similar position, we're after cheap capacity rather than performance but the smr's in our main cluster havn't been that impressive.
[0:15] <mfa298_> We do have a lot per storage box though which may not help.
[0:16] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[0:16] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[0:16] <Thunderbird> what's the ratio between disks and 10gb/s nics? (or whatever networking you are using)
[0:17] <mfa298_> For my test clusters I have 5 storage nodes each with 45 drives per node. SMRs on their own managed 400-500MB/s but once I got blocked requests that tanked. With the SSD journals (and only a quick test) that was up to 700MB/s (maybe a bit more), a similar setup with 6TB WD greens was getting 1300MB/s without ssd journals
[0:18] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[0:18] * Icey (~IceyEC_@0001bbad.user.oftc.net) Quit (Ping timeout: 480 seconds)
[0:18] <mfa298_> test clusters have been single 10G links per node, main cluster is 2x 20G with 70x SMR drives (plus a few SSDs and 6TB drives as well)
[0:19] <mfa298_> but throughput is nowhere near that level. I think the best I managed on the main cluster is 2GBytes/s which is ~2000 drives
[0:19] <Thunderbird> I would have guessed like 12-16 drives for 10G, but I guess you are going for a lot of storage, we may aim for lots of nodes
[0:20] <mfa298_> (those cluster speeds are all Bytes/s, using our own simulations for writing data and monitoring how the data in the cluster grows over time)
[0:20] * sbfox (~Adium@vancouver.xmatters.com) Quit (Quit: Leaving.)
[0:20] * davidzlap1 (~Adium@2605:e000:1313:8003:e83b:414f:4015:1417) has joined #ceph
[0:21] <mfa298_> we've had various stability issues which I think are partly due to the number of drives per box, but also just due to the SMR drives.
[0:22] * davidzlap (~Adium@2605:e000:1313:8003:e83b:414f:4015:1417) Quit (Read error: Connection reset by peer)
[0:24] <mfa298_> I did try some node failure tests against the two test clusters, putting the same amount of data on to the clusters and failing one node. In a week on the 6TB cluster we failed a node and had it rebalance, did a crush remove on the down+out drives and had it rebalance, then bring the node back online and rebalance.
[0:24] <mfa298_> In the same time the SMR cluster hadn't finished the initial node down rebalance
[0:26] * vata (~vata@207.96.182.162) Quit (Ping timeout: 480 seconds)
[0:26] <mfa298_> we do have recovery configured to a low level so that we don't impact normal cluster operations, but both clusters had the same configs applied.
[0:27] * sileht (~sileht@sileht.net) Quit (Ping timeout: 480 seconds)
[0:29] <mfa298_> I'm hoping as updates happen to ext4/xfs the smr drives will improve (although most of those seem to be aimed at Host Aware and Host managed rather than Drive managed)
[0:30] <diq_> hey thinking about removing SPOFs and avoiding things like HA proxy, are there any downsides to running the S3 gateway on over OSD node? Not 1 per OSD obviously, but 1 per OSD server.
[0:30] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[0:30] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[0:33] * lcurtis (~lcurtis@47.19.105.250) Quit (Remote host closed the connection)
[0:33] <diq_> Sorry I meant 1 per every OSD node (over was a typo)
[0:35] <Thunderbird> mfa298_, only for host managed drives I expect things to improve, drive managed ones are basically like SSDs on the inside in terms of garbage collection, there isn't much the host can do (I guess trim helps)
[0:35] * TheArchivist (~root@184.68.16.194) has joined #ceph
[0:37] <TheArchivist> Anybody want to help me get my cluster going?
[0:38] <mfa298_> Thunderbird: agreed, there doesn't seem to be much scope for real improvements, but I live in hope some of the changes might help the drive managed drives do their job better.
[0:39] <Thunderbird> TheArchivist, first tip don't irc as root
[0:39] <TheArchivist> aw shit...... fucking noob mistake....
[0:39] <TheArchivist> exit
[0:39] * TheArchivist (~root@184.68.16.194) Quit (Quit: leaving)
[0:40] * TheArchivist (~joshua@184.68.16.194) has joined #ceph
[0:40] <zao> Thunderbird: I love getting those friendly tips when doing IRC over DNS/HTTP tunnels.
[0:40] * angdraug (~angdraug@12.164.168.117) Quit (Ping timeout: 480 seconds)
[0:40] <mfa298_> I suspect by the time the changes get to a sensibly useable state things will have moved on anyway (I don't really fancy having to deploy custom kernels and user space tools everywhere)
[0:40] <zao> ident ends up being root@ on the tunnel machine
[0:41] * sbfox (~Adium@vancouver.xmatters.com) has joined #ceph
[0:42] <TheArchivist> Believe it or not, I've been using linux since 0.99pl12. Fsck... My firewall is filtering 113.
[0:43] * debian112 (~bcolbert@24.126.201.64) Quit (Quit: Leaving.)
[0:45] <TheArchivist> So is 0.94.5 in a usable state? I'm deploying to 10 14.04.3 nodes.
[0:47] * sbfox (~Adium@vancouver.xmatters.com) Quit (Quit: Leaving.)
[0:47] <diq_> I've run into more issues with hardware and appropriate tuning than I have the actual ceph software
[0:48] <TheArchivist> Everything is dual or triple core, most with virtualization, dual gig networks.
[0:48] <Thunderbird> depending on your use case I/O could be a little weak, but it should work
[0:48] <diq_> depending on your drive/node density you will want more network juice
[0:49] * stiopa (~stiopa@cpc73828-dals21-2-0-cust630.20-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[0:49] <diq_> have you timed how long it takes to replicate a failed node under load? With scrubbing?
[0:49] <TheArchivist> This is a test network. I'm hoping I can get ceph-dokan and fuse under OpenBSD to work.
[0:50] <diq_> I went through some ceph testing with lower grade hardware. It was frustrating. Test on what you'll run in production (or as close to it as you can get)
[0:51] <TheArchivist> pffft. I am self employed.
[0:52] * angdraug (~angdraug@12.164.168.117) has joined #ceph
[0:52] <mfa298_> I'd definetly agree with diq_ on testing with what you plan to run in production, our early tests were on some woefully underpowered boxes and we could never get it stable
[0:52] <TheArchivist> Production comes in when the customer is happy that ceph works at all in his setting. I am shouldering the costs for this.
[0:53] <diq_> that's my fear. My underpowered testing actually wrecked the cluster stability.
[0:53] * dgbaley27 (~matt@c-67-176-93-83.hsd1.co.comcast.net) Quit (Quit: Leaving.)
[0:54] <TheArchivist> I am only testing with 500G drives for data, 80GB for journal, and 80GB for OS. From the specs on the website, I should be good, or so it says.
[0:55] * ircolle (~Adium@2601:285:201:2bf9:8af:2fa5:450b:60dc) Quit (Quit: Leaving.)
[0:57] <mfa298_> VM's are good for initial testing to see how it works and to try breaking it, but things will likely be different when you move to real hardware.
[0:58] <mfa298_> how many of your 500G drives are sharing that 80G journal ?
[0:58] <TheArchivist> I'm not testing on VMs. I have 10 boxes with 4-6GB RAM, a journal drive, a data drive, and an OS drive.
[0:59] <TheArchivist> journal drive is 80GB, data is 500GB, OS is 80GB.
[1:00] <diq_> while definitely not apples to apples, RedHat has a sizing guide for their storage system based on Ceph
[1:00] <TheArchivist> ceph-deploy leaves crap on my filesystems when I ceph-deploy purge.
[1:00] <diq_> so many ghz per OSD
[1:00] <diq_> so much RAM per OSD
[1:00] <mfa298_> ah I misunderstood, I thought you were implying it was inside VMs, 80G journal is probably massivly over specced
[1:00] <TheArchivist> I get them for $5.
[1:01] <Thunderbird> TheArchivist, use purgedata, though I'm not a fan of ceph-deploy
[1:02] <TheArchivist> Sorry... I meant purgedata....
[1:02] <mfa298_> you could probably run a few data drives per box on that setup, although you might want to bump the ram up a bit if you do that.
[1:02] <Thunderbird> it just removes /etc/ceph and /var/lib/ceph
[1:02] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[1:02] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[1:03] <TheArchivist> Okay... so if ceph-deploy sucks, what's better? The Ansible script? Chef?
[1:03] <TheArchivist> Yeah, and I have noticed that ceph-deploy purge leaves daemons running. Pretty shoddy.
[1:04] <Thunderbird> no experience with the other methods other than manual, because most are unusable in my environment
[1:04] <mfa298_> I just ran the few commands that ceph-deploy does manually, once you've got the monitors up and push the bootstrap keys around there's not much to do.
[1:05] <mfa298_> for testing I've not got a couple of scripts to teardown the cluster and build a new one which just takes a few changes to do slightly different things
[1:05] <TheArchivist> Thank you all for the hardware recommendations, but what I have said is what I have in the way of hardware. I've seen thinner deployment in VM tests.
[1:06] <Thunderbird> now having the joy of figuring out some ceph authentication errors, just 'ceph status' claims invalid argument internally; I'm passing in my config file and keys on the command line
[1:07] <Thunderbird> if I pass in the monitor key (which should be wrong) I get EACCESS, but I get the invalid argument error when using the client admin key
[1:08] <Thunderbird> main error in the monitor log 'v2 caught error when trying to handle auth request, probably malformed request'
[1:09] <TheArchivist> Is it just me, or is the documentation on the site excremental? The ceph-deploy route fails when I try to set up initial mons.
[1:10] * Icey (~IceyEC_@0001bbad.user.oftc.net) has joined #ceph
[1:12] * kefu (~kefu@114.92.106.70) has joined #ceph
[1:15] * fsimonce (~simon@host30-173-dynamic.23-79-r.retail.telecomitalia.it) Quit (Remote host closed the connection)
[1:17] * bene2 (~bene@2601:18c:8300:f3ae:ea2a:eaff:fe08:3c7a) Quit (Quit: Konversation terminated!)
[1:19] * Icey (~IceyEC_@0001bbad.user.oftc.net) Quit (Ping timeout: 480 seconds)
[1:19] * TheArchivist (~joshua@184.68.16.194) Quit (Quit: leaving)
[1:20] * dupont-y (~dupont-y@familledupont.org) Quit (Ping timeout: 480 seconds)
[1:22] * gregmark (~Adium@68.87.42.115) Quit (Quit: Leaving.)
[1:26] * vata (~vata@cable-21.246.173-197.electronicbox.net) has joined #ceph
[1:27] * rendar (~I@host225-177-dynamic.21-87-r.retail.telecomitalia.it) Quit (Quit: std::lower_bound + std::less_equal *works* with a vector without duplicates!)
[1:27] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[1:27] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[1:29] * MentalRay (~MentalRay@107.171.161.165) has joined #ceph
[1:29] * dgurtner_ (~dgurtner@185.10.235.250) Quit (Ping timeout: 480 seconds)
[1:31] * Gorazd (~Gorazd@89-212-99-37.dynamic.t-2.net) Quit (Ping timeout: 480 seconds)
[1:33] * jwilkins (~jowilkin@67.204.149.211) Quit (Ping timeout: 480 seconds)
[1:33] * davidzlap (~Adium@2605:e000:1313:8003:e83b:414f:4015:1417) has joined #ceph
[1:33] * davidzlap1 (~Adium@2605:e000:1313:8003:e83b:414f:4015:1417) Quit (Read error: Connection reset by peer)
[1:47] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[1:47] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[1:52] * georgem (~Adium@69-196-163-180.dsl.teksavvy.com) has joined #ceph
[1:54] * georgem (~Adium@69-196-163-180.dsl.teksavvy.com) Quit ()
[1:58] * sudocat (~dibarra@192.185.1.20) Quit (Remote host closed the connection)
[2:07] * georgem (~Adium@69-196-163-180.dsl.teksavvy.com) has joined #ceph
[2:08] * Steki (~steki@cable-89-216-234-219.dynamic.sbb.rs) has joined #ceph
[2:11] * Icey (~IceyEC_@0001bbad.user.oftc.net) has joined #ceph
[2:11] * georgem (~Adium@69-196-163-180.dsl.teksavvy.com) Quit ()
[2:15] <diq_> the documentation sometimes doesn't keep up with releases
[2:15] * BManojlovic (~steki@cable-89-216-194-159.dynamic.sbb.rs) Quit (Ping timeout: 480 seconds)
[2:15] * red (~red@chello084112110034.11.11.vie.surfer.at) has joined #ceph
[2:15] <diq_> there will be steps missing or certain steps that are no longer required
[2:15] <diq_> oh he left
[2:15] <olid14> ^^;
[2:16] <diq_> but hey IRC is living documentation as long as someone with a clue is paying attention :D
[2:16] <olid14> yeah, the big IF
[2:17] <olid14> for me the mailing list archive was very helpful
[2:21] * Icey (~IceyEC_@0001bbad.user.oftc.net) Quit (Ping timeout: 480 seconds)
[2:22] * redf_ (~red@chello084112110034.11.11.vie.surfer.at) Quit (Ping timeout: 480 seconds)
[2:23] * georgem (~Adium@69-196-163-180.dsl.teksavvy.com) has joined #ceph
[2:23] * georgem (~Adium@69-196-163-180.dsl.teksavvy.com) Quit ()
[2:24] * georgem (~Adium@69-196-163-180.dsl.teksavvy.com) has joined #ceph
[2:25] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[2:25] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[2:25] * angdraug (~angdraug@12.164.168.117) Quit (Quit: Leaving)
[2:31] * zhaochao (~zhaochao@125.39.8.237) has joined #ceph
[2:32] * davidzlap1 (~Adium@2605:e000:1313:8003:e83b:414f:4015:1417) has joined #ceph
[2:33] * davidzlap (~Adium@2605:e000:1313:8003:e83b:414f:4015:1417) Quit (Read error: Connection reset by peer)
[2:35] * cholcombe (~chris@c-73-180-29-35.hsd1.or.comcast.net) Quit (Ping timeout: 480 seconds)
[2:38] * lightspeed (~lightspee@fw-carp-wan.ext.lspeed.org) Quit (Ping timeout: 480 seconds)
[2:41] * olid14 (~olid1982@185.17.206.44) Quit (Ping timeout: 480 seconds)
[2:42] * segutier (~segutier@sfo-vpn1.shawnlower.net) Quit (Ping timeout: 480 seconds)
[2:47] * lightspeed (~lightspee@fw-carp-wan.ext.lspeed.org) has joined #ceph
[2:47] * georgem (~Adium@69-196-163-180.dsl.teksavvy.com) Quit (Quit: Leaving.)
[2:52] * georgem (~Adium@206.108.127.16) has joined #ceph
[3:05] * Kupo1 (~tyler.wil@23.111.254.159) Quit (Read error: Connection reset by peer)
[3:06] <tacticus> diq_: it's disturbing how often i go through my irc logs for documentation :|
[3:07] * sankarshan (~sankarsha@121.244.87.117) has joined #ceph
[3:09] * naoto (~naotok@2401:bd00:b001:8920:27:131:11:254) has joined #ceph
[3:10] * kefu is now known as kefu|afk
[3:10] * fdmanana (~fdmanana@2001:8a0:6dfd:6d01:c11e:ac88:f68b:f7e8) Quit (Ping timeout: 480 seconds)
[3:10] * JCL (~JCL@ip68-108-16-17.lv.lv.cox.net) Quit (Quit: Leaving.)
[3:13] * Icey (~IceyEC_@0001bbad.user.oftc.net) has joined #ceph
[3:14] * dgbaley27 (~matt@c-67-176-93-83.hsd1.co.comcast.net) has joined #ceph
[3:17] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[3:17] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[3:20] * yanzheng (~zhyan@182.139.23.79) has joined #ceph
[3:21] * Icey (~IceyEC_@0001bbad.user.oftc.net) Quit (Ping timeout: 480 seconds)
[3:22] * jdillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) has joined #ceph
[3:25] * JCL (~JCL@ip68-108-16-17.lv.lv.cox.net) has joined #ceph
[3:31] * JCL (~JCL@ip68-108-16-17.lv.lv.cox.net) Quit (Quit: Leaving.)
[3:41] * naoto_ (~naotok@27.131.11.254) has joined #ceph
[3:45] * naoto (~naotok@2401:bd00:b001:8920:27:131:11:254) Quit (Ping timeout: 480 seconds)
[3:50] * Spessu (~Nanobot@7V7AAAZMQ.tor-irc.dnsbl.oftc.net) has joined #ceph
[3:54] * haomaiwang (~haomaiwan@li414-102.members.linode.com) has joined #ceph
[4:00] * shaunm (~shaunm@208.102.161.229) Quit (Ping timeout: 480 seconds)
[4:01] * haomaiwang (~haomaiwan@li414-102.members.linode.com) Quit (Remote host closed the connection)
[4:01] * haomaiwang (~haomaiwan@li414-102.members.linode.com) has joined #ceph
[4:04] * kefu|afk is now known as kefu
[4:04] * zao_ (~zao@h-38-154.a258.priv.bahnhof.se) has joined #ceph
[4:05] * _28_ria (~kvirc@194.226.6.106) Quit (Read error: Connection reset by peer)
[4:05] * zao (~zao@h-38-154.a258.priv.bahnhof.se) Quit (Remote host closed the connection)
[4:05] * mattronix (~quassel@2a02:898:162:1::1) Quit (Quit: No Ping reply in 180 seconds.)
[4:05] * Be-El (~quassel@fb08-bcf-pc01.computational.bio.uni-giessen.de) Quit (Remote host closed the connection)
[4:06] * mattronix (~quassel@2a02:898:162:1::1) has joined #ceph
[4:06] * adun153 (~ljtirazon@112.198.78.80) has joined #ceph
[4:06] * Be-El (~quassel@fb08-bcf-pc01.computational.bio.uni-giessen.de) has joined #ceph
[4:07] * lurbs (user@uber.geek.nz) Quit (Remote host closed the connection)
[4:07] * lurbs (user@uber.geek.nz) has joined #ceph
[4:08] * haomaiwang (~haomaiwan@li414-102.members.linode.com) Quit (Remote host closed the connection)
[4:09] * haomaiwang (~haomaiwan@45.32.13.54) has joined #ceph
[4:09] * sep (~sep@95.62-50-191.enivest.net) Quit (Ping timeout: 480 seconds)
[4:10] * sep (~sep@95.62-50-191.enivest.net) has joined #ceph
[4:10] * marcan (marcan@marcansoft.com) Quit (Remote host closed the connection)
[4:10] * marcan (marcan@marcansoft.com) has joined #ceph
[4:12] * joshd (~jdurgin@68-119-140-18.dhcp.ahvl.nc.charter.com) Quit (Quit: Leaving.)
[4:13] * sasha1 (~achuzhoy@BURLON0309W-LP140-03-1279478798.dsl.bell.ca) Quit (Ping timeout: 480 seconds)
[4:14] * Icey (~IceyEC_@0001bbad.user.oftc.net) has joined #ceph
[4:16] * CheKoLyN (~saguilar@bender.parc.xerox.com) Quit (Quit: Leaving)
[4:18] * sileht (~sileht@sileht.net) has joined #ceph
[4:20] * Spessu (~Nanobot@7V7AAAZMQ.tor-irc.dnsbl.oftc.net) Quit ()
[4:20] <m0zes> source diving. of course I like to read code.
[4:22] * Icey (~IceyEC_@0001bbad.user.oftc.net) Quit (Ping timeout: 480 seconds)
[4:25] * sasha1 (~achuzhoy@BURLON0309W-LP140-03-1279478798.dsl.bell.ca) has joined #ceph
[4:30] * haomaiwang (~haomaiwan@45.32.13.54) Quit (Remote host closed the connection)
[4:30] * overclk_ (~overclk@59.93.66.233) has joined #ceph
[4:32] * jdillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) Quit (Quit: jdillaman)
[4:45] * vbellur (~vijay@122.172.251.132) Quit (Ping timeout: 480 seconds)
[4:45] * georgem (~Adium@206.108.127.16) Quit (Quit: Leaving.)
[4:47] <adun153> Hi! Is there official documentation on how to administer a ceph cluster? Something like creating pools, resizing pools, check for usage statistics, free space left, etc.
[4:55] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[4:55] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[4:56] * vbellur (~vijay@122.167.150.126) has joined #ceph
[5:00] * kefu is now known as kefu|afk
[5:00] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit (Quit: Leaving.)
[5:04] <m0zes> adun153: something like this? http://docs.ceph.com/docs/master/rados/operations/
[5:05] <adun153> m0zes: Looks like that is it. Thanks!!!!
[5:05] * Vacuum_ (~Vacuum@i59F79D01.versanet.de) has joined #ceph
[5:10] * kefu|afk (~kefu@114.92.106.70) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[5:12] * Vacuum__ (~Vacuum@88.130.220.142) Quit (Ping timeout: 480 seconds)
[5:15] * Icey (~IceyEC_@0001bbad.user.oftc.net) has joined #ceph
[5:23] * Icey (~IceyEC_@0001bbad.user.oftc.net) Quit (Ping timeout: 480 seconds)
[5:24] * kanagaraj (~kanagaraj@121.244.87.117) has joined #ceph
[5:26] * zhaochao (~zhaochao@125.39.8.237) Quit (Quit: ChatZilla 0.9.92 [Iceweasel 38.3.0/20150922225347])
[5:39] * overclk (~overclk@59.93.65.193) has joined #ceph
[5:39] * overclk (~overclk@59.93.65.193) Quit (autokilled: This host may be infected. Mail support@oftc.net with questions. BOPM (2015-11-04 04:39:48))
[5:43] * kefu (~kefu@114.92.106.70) has joined #ceph
[5:45] * overclk_ (~overclk@59.93.66.233) Quit (Ping timeout: 480 seconds)
[5:48] * davidzlap (~Adium@2605:e000:1313:8003:e83b:414f:4015:1417) has joined #ceph
[5:49] * davidzlap1 (~Adium@2605:e000:1313:8003:e83b:414f:4015:1417) Quit (Read error: Connection reset by peer)
[5:58] * rakeshgm (~rakesh@106.51.30.186) has joined #ceph
[5:58] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[5:58] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[6:00] * vbellur (~vijay@122.167.150.126) Quit (Ping timeout: 480 seconds)
[6:01] * amote (~amote@121.244.87.116) has joined #ceph
[6:05] * kefu is now known as kefu|afk
[6:06] * kefu|afk (~kefu@114.92.106.70) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[6:06] * Mattress (~Heliwr@89.248.172.45) has joined #ceph
[6:09] * wayneeseguin (~wayneeseg@mp64.overnothing.com) Quit (Ping timeout: 480 seconds)
[6:13] * kraken (~kraken@gw.sepia.ceph.com) Quit (Ping timeout: 480 seconds)
[6:14] * tenshi (~David@MTRLPQ42-1176054809.sdsl.bell.ca) Quit (Read error: Connection reset by peer)
[6:15] * cephalobot (~ceph@ds3553.dreamservers.com) Quit (Ping timeout: 480 seconds)
[6:15] * rturk-away (~rturk@ds3553.dreamservers.com) Quit (Ping timeout: 480 seconds)
[6:16] * Icey (~IceyEC_@0001bbad.user.oftc.net) has joined #ceph
[6:24] * Icey (~IceyEC_@0001bbad.user.oftc.net) Quit (Ping timeout: 480 seconds)
[6:28] * davidzlap (~Adium@2605:e000:1313:8003:e83b:414f:4015:1417) Quit (Read error: Connection reset by peer)
[6:31] * vbellur (~vijay@121.244.87.117) has joined #ceph
[6:31] * haomaiwang (~haomaiwan@s16.ezlink.hk) has joined #ceph
[6:32] * wayneeseguin (~wayneeseg@mp64.overnothing.com) has joined #ceph
[6:32] * davidzlap (~Adium@cpe-172-91-154-245.socal.res.rr.com) has joined #ceph
[6:36] * Mattress (~Heliwr@7V7AAAZOW.tor-irc.dnsbl.oftc.net) Quit ()
[6:38] * overclk (~overclk@59.93.64.182) has joined #ceph
[6:45] * nwf (~nwf@00018577.user.oftc.net) Quit (Remote host closed the connection)
[6:45] * nwf (~nwf@00018577.user.oftc.net) has joined #ceph
[6:51] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[6:51] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[6:53] * kefu (~kefu@114.92.106.70) has joined #ceph
[7:01] * haomaiwang (~haomaiwan@s16.ezlink.hk) Quit (Remote host closed the connection)
[7:01] * vbellur (~vijay@121.244.87.117) Quit (Ping timeout: 480 seconds)
[7:01] * haomaiwang (~haomaiwan@s16.ezlink.hk) has joined #ceph
[7:01] * karnan (~karnan@121.244.87.117) has joined #ceph
[7:02] * MentalRay (~MentalRay@107.171.161.165) Quit (Quit: This computer has gone to sleep)
[7:04] * kefu (~kefu@114.92.106.70) Quit (Max SendQ exceeded)
[7:05] * kefu (~kefu@114.92.106.70) has joined #ceph
[7:10] * stein (~stein@185.56.185.82) Quit (Remote host closed the connection)
[7:14] * vbellur (~vijay@121.244.87.124) has joined #ceph
[7:14] * cephalobot (~ceph@ds3553.dreamservers.com) has joined #ceph
[7:15] * rturk-away (~rturk@ds3553.dreamservers.com) has joined #ceph
[7:16] * Icey (~IceyEC_@0001bbad.user.oftc.net) has joined #ceph
[7:25] * Icey (~IceyEC_@0001bbad.user.oftc.net) Quit (Ping timeout: 480 seconds)
[7:26] * stein (~stein@185.56.185.82) has joined #ceph
[7:29] * DV (~veillard@2001:41d0:a:f29f::1) Quit (Ping timeout: 480 seconds)
[7:30] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[7:30] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[7:39] * kawa2014 (~kawa@94.160.31.42) has joined #ceph
[7:42] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[7:42] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[7:47] * kawa2014 (~kawa@94.160.31.42) Quit (Ping timeout: 480 seconds)
[7:48] * kawa2014 (~kawa@94.166.207.101) has joined #ceph
[7:51] * Xeon06 (~luigiman@h-213.61.149.100.host.de.colt.net) has joined #ceph
[8:01] * haomaiwang (~haomaiwan@s16.ezlink.hk) Quit (Remote host closed the connection)
[8:01] * haomaiwang (~haomaiwan@s16.ezlink.hk) has joined #ceph
[8:15] * nardial (~ls@dslb-178-009-201-039.178.009.pools.vodafone-ip.de) has joined #ceph
[8:17] * Icey (~IceyEC_@0001bbad.user.oftc.net) has joined #ceph
[8:21] * Xeon06 (~luigiman@5P6AAASTJ.tor-irc.dnsbl.oftc.net) Quit ()
[8:25] * Icey (~IceyEC_@0001bbad.user.oftc.net) Quit (Ping timeout: 480 seconds)
[8:27] * dugravot6 (~dugravot6@dn-infra-04.lionnois.univ-lorraine.fr) has joined #ceph
[8:27] * lookcrabs (~lookcrabs@tail.seeee.us) Quit (Ping timeout: 480 seconds)
[8:29] * shinobu (~oftc-webi@nat-pool-nrt-t1.redhat.com) has joined #ceph
[8:29] * derjohn_mob (~aj@p578b6aa1.dip0.t-ipconnect.de) has joined #ceph
[8:30] * Bobby (~Borf@se3x.mullvad.net) has joined #ceph
[8:32] * madkiss (~madkiss@2001:6f8:12c3:f00f:b9a0:8ede:9404:feb0) Quit (Remote host closed the connection)
[8:40] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[8:40] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[8:40] * fdmanana (~fdmanana@2001:8a0:6dfd:6d01:c11e:ac88:f68b:f7e8) has joined #ceph
[8:41] * fridim_ (~fridim@56-198-190-109.dsl.ovh.fr) has joined #ceph
[8:45] * nardial (~ls@dslb-178-009-201-039.178.009.pools.vodafone-ip.de) Quit (Ping timeout: 480 seconds)
[8:48] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[8:48] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[8:49] <flaf> Hi, I have a question concerning Infernalis. Now, the daemons should use the ceph unix account and of course the working dir must be owned by ceph (of instance /var/lib/ceph/osd/ceph-$id/ etc.)
[8:50] * dgurtner (~dgurtner@nat-pool-ams2-wlan-u.redhat.com) has joined #ceph
[8:50] <flaf> But in my case /var/lib/ceph/osd/ceph-$id/journal is a symlink to a device /dev/sdXY.
[8:50] <flaf> And this device have to be owned by ceph too.
[8:50] <flaf> *has
[8:51] <flaf> the ceph account need to write in the journal.
[8:52] <flaf> But it's a little bit complicated to manage the unix rights of a device /dev/sdXY because after a reboot the unix rights arre deleted.
[8:54] <flaf> How do you handle the Unix rights of the journal when the journal is a symlink to a device in /dev?
[8:54] * bara (~bara@ip4-83-240-10-82.cust.nbox.cz) has joined #ceph
[8:58] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[8:58] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[8:59] * mario7 (~mario@46.77.124.86) has joined #ceph
[9:00] * ibravo (~ibravo@72.83.69.64) Quit (Quit: Leaving)
[9:00] * Bobby (~Borf@4Z9AAASFH.tor-irc.dnsbl.oftc.net) Quit ()
[9:01] * haomaiwang (~haomaiwan@s16.ezlink.hk) Quit (Remote host closed the connection)
[9:01] * haomaiwang (~haomaiwan@s16.ezlink.hk) has joined #ceph
[9:02] <mario7> hello, thinking of implementing cepf with erasure code, is this the right place to ask some questions?
[9:06] * jasuarez (~jasuarez@186.Red-83-37-101.dynamicIP.rima-tde.net) has joined #ceph
[9:07] <yanzheng> flaf, just change owner of /dev/sdxy
[9:07] <yanzheng> udder rule can do that
[9:07] <yanzheng> udev rule
[9:07] <flaf> yanzheng: yes but as I said after a reboot the unix right are "deleted"
[9:08] <yanzheng> http://www.reactivated.net/writing_udev_rules.html
[9:09] * dugravot61 (~dugravot6@nat-persul-plg.wifi.univ-lorraine.fr) has joined #ceph
[9:09] <flaf> Ok I have to use udev rules to ensure the owner. I thought indeed that it was the solution. Thx.
[9:09] * dugravot6 (~dugravot6@dn-infra-04.lionnois.univ-lorraine.fr) Quit (Ping timeout: 480 seconds)
[9:09] * rendar (~I@host177-119-dynamic.57-82-r.retail.telecomitalia.it) has joined #ceph
[9:10] * analbeard (~shw@support.memset.com) has joined #ceph
[9:12] * mario7 (~mario@46.77.124.86) Quit (Quit: Leaving)
[9:15] * fsimonce (~simon@host30-173-dynamic.23-79-r.retail.telecomitalia.it) has joined #ceph
[9:18] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[9:18] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[9:18] * Icey (~IceyEC_@0001bbad.user.oftc.net) has joined #ceph
[9:20] * nardial (~ls@ipservice-092-209-179-024.092.209.pools.vodafone-ip.de) has joined #ceph
[9:20] * b0e (~aledermue@213.95.25.82) has joined #ceph
[9:20] <Be-El> flaf: debian and others assign a unix group to all disk devices ('disk' group). adding the ceph user to that group should fix the problem, too
[9:20] * pabluk_ is now known as pabluk
[9:21] * bara (~bara@ip4-83-240-10-82.cust.nbox.cz) Quit (Quit: Bye guys! (??????????????????? ?????????)
[9:22] <flaf> Be-El: ah yes indeed. I think I prefer try the udev rule because if ceph belongs to the disk group, he can write one every disk.
[9:22] <Be-El> flaf: good point
[9:23] <flaf> I'm trying to find udev rule based on the partlabel. I don't know if it exist.
[9:25] <flaf> In my case, each journal has this type of name "/dev/disk/by-partlabel/osd-xx-journal" so I would like to find a generic udev rule.
[9:26] * Icey (~IceyEC_@0001bbad.user.oftc.net) Quit (Ping timeout: 480 seconds)
[9:30] * dugravot61 (~dugravot6@nat-persul-plg.wifi.univ-lorraine.fr) Quit (Quit: Leaving.)
[9:32] * branto1 (~branto@ip-213-220-232-132.net.upcbroadband.cz) has joined #ceph
[9:33] * overclk (~overclk@59.93.64.182) Quit (Remote host closed the connection)
[9:33] <Be-El> flaf: http://ubuntuforums.org/showthread.php?t=784677
[9:34] <Be-El> flaf: add a regex match and (based on my limited knowledge about udev) it might do it
[9:35] <flaf> Be-El: thx. Indeed, It seems to me that a little regex it possible in udev.
[9:35] <flaf> If I find something that work, I post it here.
[9:35] * enax (~enax@hq.ezit.hu) has joined #ceph
[9:36] * rotbeard (~redbeard@2a02:908:df11:4280:6267:20ff:feb7:c20) has joined #ceph
[9:36] * nardial (~ls@ipservice-092-209-179-024.092.209.pools.vodafone-ip.de) Quit (Quit: Leaving)
[9:39] * fridim_ (~fridim@56-198-190-109.dsl.ovh.fr) Quit (Remote host closed the connection)
[9:42] * dgbaley27 (~matt@c-67-176-93-83.hsd1.co.comcast.net) Quit (Quit: Leaving.)
[9:43] * bjozet (~bjozet@82-183-17-144.customers.ownit.se) has joined #ceph
[9:43] * rotbeard (~redbeard@2a02:908:df11:4280:6267:20ff:feb7:c20) Quit (Quit: Leaving)
[9:43] * rotbeard (~redbeard@2a02:908:df11:4280:6267:20ff:feb7:c20) has joined #ceph
[9:46] <Be-El> flaf: o you use gpt partition tables? if that is the case, you can also match on the partition type uuid, similar to the existing udev rules
[9:46] <Be-El> +d
[9:46] <flaf> Excalty. I'm trying currently. ;)
[9:49] * bitserker (~toni@88.87.194.130) has joined #ceph
[9:49] * fsimonce (~simon@host30-173-dynamic.23-79-r.retail.telecomitalia.it) Quit (Quit: Coyote finally caught me)
[9:53] * garphy`aw is now known as garphy
[9:53] * thomnico (~thomnico@2a01:e35:8b41:120:a966:fe0a:1aa3:4334) has joined #ceph
[9:58] * sankarshan (~sankarsha@121.244.87.117) Quit (Quit: Are you sure you want to quit this channel (Cancel/Ok) ?)
[9:59] * jordanP (~jordan@204.13-14-84.ripe.coltfrance.com) has joined #ceph
[9:59] * overclk (~overclk@59.93.64.182) has joined #ceph
[10:01] * haomaiwang (~haomaiwan@s16.ezlink.hk) Quit (Remote host closed the connection)
[10:01] <flaf> Be-El: In fact, it's short https://gist.github.com/flaf/e64c3bd97e5353119643 (tested)
[10:01] * bara (~bara@213.175.37.12) has joined #ceph
[10:01] * haomaiwang (~haomaiwan@45.32.13.54) has joined #ceph
[10:02] <flaf> I'm a doubt concerning the name of the file (/etc/udev/rules.d/90-ceph.rules). I don't known if the number is really important...
[10:02] <kiranos> its the order they are loaded
[10:03] <flaf> A "big" value is probably better.
[10:03] <kiranos> so if you have dependencies etc its better to load it late
[10:03] * haomaiwang (~haomaiwan@45.32.13.54) Quit (Remote host closed the connection)
[10:04] * fsimonce (~simon@host30-173-dynamic.23-79-r.retail.telecomitalia.it) has joined #ceph
[10:05] <Be-El> flaf: 90 should be fine, since you need to run it before the osd starts up (which is done by 95-ceph-osd or similar)
[10:06] <flaf> Be-El: is it udev which starts OSDs? This is not systemd/upstart etc. ?
[10:07] <Be-El> flaf: at least in ubuntu trusty it's udev
[10:07] <flaf> I thought it was upstart.
[10:07] <Be-El> flaf: there's also a call to ceph-disk activate-all as upstart script, but i'm not sure whether it actually starts the osds
[10:08] <flaf> ok, me too. If it's udev, it seems to me very curieux.
[10:08] <flaf> *curious.
[10:08] * dgurtner (~dgurtner@nat-pool-ams2-wlan-u.redhat.com) Quit (Ping timeout: 480 seconds)
[10:08] * T1w (~jens@node3.survey-it.dk) has joined #ceph
[10:10] * haomaiwang (~haomaiwan@s16.ezlink.hk) has joined #ceph
[10:11] * haomaiwang (~haomaiwan@s16.ezlink.hk) Quit (Remote host closed the connection)
[10:11] * olid14 (~olid1982@aftr-185-17-204-236.dynamic.mnet-online.de) has joined #ceph
[10:16] <sep> anyone know tricks to reduce ram usage on ceph-osd's ? my test cluster died horribly when OOM killer started killing osd processes all over. i have way to little ram in them, but it's the hardware i have.
[10:16] <sep> quite willing to sacrifice performance for stabillity.
[10:18] * adun153 (~ljtirazon@112.198.78.80) Quit (Read error: Connection reset by peer)
[10:19] * Icey (~IceyEC_@0001bbad.user.oftc.net) has joined #ceph
[10:24] * brutuscat (~brutuscat@60.Red-193-152-185.dynamicIP.rima-tde.net) has joined #ceph
[10:27] * Icey (~IceyEC_@0001bbad.user.oftc.net) Quit (Ping timeout: 480 seconds)
[10:31] * derjohn_mob (~aj@p578b6aa1.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[10:32] * adun153 (~ljtirazon@112.198.98.232) has joined #ceph
[10:33] * Concubidated (~Adium@tmo-111-152.customers.d1-online.com) has joined #ceph
[10:35] * shinobu (~oftc-webi@nat-pool-nrt-t1.redhat.com) Quit (Quit: Page closed)
[10:38] * joao (~joao@217.149.135.180) has joined #ceph
[10:39] * hgichon (~hgichon@220.90.135.162) Quit (Ping timeout: 480 seconds)
[10:43] * joao (~joao@217.149.135.180) Quit (Quit: leaving)
[10:46] * ngoswami (~ngoswami@121.244.87.116) has joined #ceph
[10:47] * kawa2014 (~kawa@94.166.207.101) Quit (Ping timeout: 480 seconds)
[10:51] * ade (~abradshaw@tmo-111-182.customers.d1-online.com) has joined #ceph
[10:55] * dgurtner (~dgurtner@77.95.96.78) has joined #ceph
[10:55] * olid14 (~olid1982@aftr-185-17-204-236.dynamic.mnet-online.de) Quit (Quit: Nettalk6 - www.ntalk.de)
[10:55] * derjohn_mob (~aj@fw.gkh-setu.de) has joined #ceph
[10:58] * joao (~joao@217.149.135.180) has joined #ceph
[10:58] * ChanServ sets mode +o joao
[10:59] * olid14 (~olid1982@aftr-185-17-204-236.dynamic.mnet-online.de) has joined #ceph
[11:08] * CSa_ (~christian@mintzer.imp.fu-berlin.de) has joined #ceph
[11:09] * kefu is now known as kefu|afk
[11:13] * remdem (~desmonet@liche.ircam.fr) Quit (Remote host closed the connection)
[11:14] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[11:14] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[11:15] <Heebie> Does anyone else have problems with VM's that have ceph installed and use RBD's, having their NICs fail?
[11:19] * Icey (~IceyEC_@0001bbad.user.oftc.net) has joined #ceph
[11:20] * ksperis (~laurent@46.218.42.103) has joined #ceph
[11:23] * kefu|afk (~kefu@114.92.106.70) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[11:23] * DV (~veillard@2001:41d0:1:d478::1) has joined #ceph
[11:25] * DV (~veillard@2001:41d0:1:d478::1) Quit ()
[11:26] * DV (~veillard@2001:41d0:1:d478::1) has joined #ceph
[11:26] * dgurtner (~dgurtner@77.95.96.78) Quit (Ping timeout: 480 seconds)
[11:28] * Icey (~IceyEC_@0001bbad.user.oftc.net) Quit (Ping timeout: 480 seconds)
[11:28] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[11:28] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[11:28] * kefu (~kefu@114.92.106.70) has joined #ceph
[11:41] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[11:41] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[11:48] * dis is now known as Guest7634
[11:48] * dis (~dis@00018d20.user.oftc.net) has joined #ceph
[11:48] * lkoranda_ (~lkoranda@nat-pool-brq-t.redhat.com) has joined #ceph
[11:48] * bara (~bara@213.175.37.12) Quit (Read error: Connection reset by peer)
[11:49] * bara (~bara@213.175.37.12) has joined #ceph
[11:52] * fridim_ (~fridim@56-198-190-109.dsl.ovh.fr) has joined #ceph
[11:54] * dis is now known as Guest7635
[11:54] * dis (~dis@00018d20.user.oftc.net) has joined #ceph
[11:54] * analbeard (~shw@support.memset.com) Quit (Quit: Leaving.)
[11:55] * Guest7634 (~dis@00018d20.user.oftc.net) Quit (Ping timeout: 480 seconds)
[11:55] * lkoranda (~lkoranda@nat-pool-brq-t.redhat.com) Quit (Ping timeout: 480 seconds)
[11:56] * linjan_ (~linjan@176.195.215.226) has joined #ceph
[11:56] * Guest7635 (~dis@00018d20.user.oftc.net) Quit (Ping timeout: 480 seconds)
[11:56] * lkoranda_ (~lkoranda@nat-pool-brq-t.redhat.com) Quit (Ping timeout: 480 seconds)
[11:56] * lkoranda (~lkoranda@213.175.37.10) has joined #ceph
[11:58] * rotbeard (~redbeard@2a02:908:df11:4280:6267:20ff:feb7:c20) Quit (Quit: Leaving)
[11:59] * adun153 (~ljtirazon@112.198.98.232) Quit (Quit: Leaving)
[12:01] * Concubidated (~Adium@tmo-111-152.customers.d1-online.com) Quit (Quit: Leaving.)
[12:02] * Kurt (~Adium@2001:628:1:5:f5c5:104c:ce7a:27f8) Quit (Quit: Leaving.)
[12:04] * linjan__ (~linjan@176.195.239.174) Quit (Ping timeout: 480 seconds)
[12:04] * joao (~joao@217.149.135.180) Quit (Quit: leaving)
[12:04] * analbeard (~shw@support.memset.com) has joined #ceph
[12:05] * joao (~joao@8.184.114.89.rev.vodafone.pt) has joined #ceph
[12:05] * ChanServ sets mode +o joao
[12:13] * analbeard (~shw@support.memset.com) Quit (Quit: Leaving.)
[12:16] * derjohn_mob (~aj@fw.gkh-setu.de) Quit (Ping timeout: 480 seconds)
[12:18] * kawa2014 (~kawa@93-62-73-252.ip21.fastwebnet.it) has joined #ceph
[12:18] * rotbeard (~redbeard@2a02:908:df11:4280:6267:20ff:feb7:c20) has joined #ceph
[12:19] * dugravot6 (~dugravot6@dn-infra-04.lionnois.univ-lorraine.fr) has joined #ceph
[12:23] * drankis (~drankis__@89.111.13.198) has joined #ceph
[12:26] * derjohn_mob (~aj@fw.gkh-setu.de) has joined #ceph
[12:26] <loicd> jamespage: hi ! do you have access to the ceph lab ?
[12:27] <jamespage> loicd, not that I am aware of
[12:29] <jamespage> loicd, (and good morning )
[12:29] <jamespage> :-)
[12:29] <loicd> and good morning :-)
[12:29] * overclk (~overclk@59.93.64.182) Quit (Remote host closed the connection)
[12:31] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[12:31] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Remote host closed the connection)
[12:34] * Malcovent (~xolotl@7V7AAAZW1.tor-irc.dnsbl.oftc.net) has joined #ceph
[12:35] * analbeard (~shw@support.memset.com) has joined #ceph
[12:43] * naoto_ (~naotok@27.131.11.254) Quit (Quit: Leaving...)
[12:45] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[12:45] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[12:48] * karnan (~karnan@121.244.87.117) Quit (Remote host closed the connection)
[12:53] * rakeshgm (~rakesh@106.51.30.186) Quit (Quit: Leaving)
[12:57] * kefu (~kefu@114.92.106.70) Quit (Max SendQ exceeded)
[12:58] * bara (~bara@213.175.37.12) Quit (Ping timeout: 480 seconds)
[12:58] * kefu (~kefu@114.92.106.70) has joined #ceph
[12:59] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[12:59] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[13:02] * overclk (~overclk@59.93.64.182) has joined #ceph
[13:04] * Malcovent (~xolotl@7V7AAAZW1.tor-irc.dnsbl.oftc.net) Quit ()
[13:04] * kefu (~kefu@114.92.106.70) Quit (Max SendQ exceeded)
[13:07] * bara (~bara@213.175.37.10) has joined #ceph
[13:08] * T1w (~jens@node3.survey-it.dk) Quit (Ping timeout: 480 seconds)
[13:08] * shinobu (~oftc-webi@pdf874cc8.tokynt01.ap.so-net.ne.jp) has joined #ceph
[13:15] * cdelatte (~cdelatte@2606:a000:6e72:300:1cd8:fba9:7576:7310) has joined #ceph
[13:21] * Icey (~IceyEC_@0001bbad.user.oftc.net) has joined #ceph
[13:23] * tuxcrafter (~jelle@ebony.powercraft.nl) Quit (Ping timeout: 480 seconds)
[13:24] * shinobu (~oftc-webi@pdf874cc8.tokynt01.ap.so-net.ne.jp) Quit (Ping timeout: 480 seconds)
[13:26] <Anticimex> are there any known issues with rgw in 0.94.5 ?
[13:26] <Anticimex> i'm getting somewhat random failures when creating buckets
[13:27] <Anticimex> some are not possible to write to, others are
[13:28] <Anticimex> when attempting to delete one of those that got broken
[13:28] <Anticimex> 2015-11-04 13:26:41.625111 7fa997fbf700 1 -- 10.19.22.137:0/1012705 --> 10.19.22.55:6803/8237 -- osd_op(client.685777.0:1104 .dir.default.685777.5 [call rgw.bucket_list] 29.5f6a6c34 ack+read+known_if_redirected e485) v5 -- ?+0 0x7fa9c00e7010 con 0x2c1e5d0
[13:28] <Anticimex> 2015-11-04 13:26:41.626439 7fa9d84f5700 1 -- 10.19.22.137:0/1012705 <== osd.15 10.19.22.55:6803/8237 34 ==== osd_op_reply(1104 .dir.default.685777.5 [call] v0'0 uv0 ondisk = -22 ((22) Invalid argument)) v6 ==== 188+0+0 (1281360630 0 0) 0x7fa9bc05ff80 con 0x2c1e5d0
[13:28] * kanagaraj (~kanagaraj@121.244.87.117) Quit (Quit: Leaving)
[13:28] <Anticimex> 2015-11-04 13:26:41.626585 7fa997fbf700 1 -- 10.19.22.137:0/1012705 --> 10.19.22.53:6802/24488 -- osd_op(client.685777.0:1105 2015-11-04-13-default.685777.5-bucket4 [append 0~192] 31.7ac3c052 ack+ondisk+write+known_if_redirected e485) v5 -- ?+0 0x7fa9c00e7870 con 0x2c53df0
[13:28] <Anticimex> invalid arg?
[13:29] * dgurtner (~dgurtner@77.95.96.78) has joined #ceph
[13:29] * Icey (~IceyEC_@0001bbad.user.oftc.net) Quit (Ping timeout: 480 seconds)
[13:30] <Anticimex> rgw running ceph 0.94.5, osds running 0.94.2
[13:30] <Anticimex> upgrading spinners now :]
[13:33] * lpabon (~quassel@24-151-54-34.dhcp.nwtn.ct.charter.com) has joined #ceph
[13:38] * tenshi (~David@MTRLPQ42-1176054809.sdsl.bell.ca) has joined #ceph
[13:38] * dgurtner (~dgurtner@77.95.96.78) Quit (Ping timeout: 480 seconds)
[13:41] * haomaiwang (~haomaiwan@s16.ezlink.hk) has joined #ceph
[13:43] * georgem (~Adium@184.151.178.173) has joined #ceph
[13:44] <Anticimex> seems to have helped
[13:47] * ninkotech (~duplo@static-84-242-87-186.net.upcbroadband.cz) has joined #ceph
[13:55] * tuxcrafter (~jelle@ebony.powercraft.nl) has joined #ceph
[14:01] * haomaiwang (~haomaiwan@s16.ezlink.hk) Quit (Remote host closed the connection)
[14:01] * haomaiwang (~haomaiwan@s16.ezlink.hk) has joined #ceph
[14:02] * vbellur (~vijay@121.244.87.124) Quit (Ping timeout: 480 seconds)
[14:03] * rwheeler (~rwheeler@pool-173-48-214-9.bstnma.fios.verizon.net) Quit (Quit: Leaving)
[14:16] * overclk (~overclk@59.93.64.182) Quit (Remote host closed the connection)
[14:22] * Icey (~IceyEC_@0001bbad.user.oftc.net) has joined #ceph
[14:23] * jdillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) has joined #ceph
[14:28] * georgem (~Adium@184.151.178.173) Quit (Quit: Leaving.)
[14:30] * Icey (~IceyEC_@0001bbad.user.oftc.net) Quit (Ping timeout: 480 seconds)
[14:33] * shaunm (~shaunm@208.102.161.229) has joined #ceph
[14:41] * xarses (~xarses@118.103.8.153) has joined #ceph
[14:46] * overclk (~overclk@59.93.64.182) has joined #ceph
[14:54] * georgem (~Adium@206.108.127.16) has joined #ceph
[14:54] * overclk (~overclk@59.93.64.182) Quit (Ping timeout: 480 seconds)
[15:00] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[15:00] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[15:01] * haomaiwang (~haomaiwan@s16.ezlink.hk) Quit (Remote host closed the connection)
[15:01] * haomaiwang (~haomaiwan@s16.ezlink.hk) has joined #ceph
[15:04] * mhack (~mhack@66-168-117-78.dhcp.oxfr.ma.charter.com) has joined #ceph
[15:06] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[15:06] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[15:06] * overclk (~overclk@59.93.64.182) has joined #ceph
[15:07] * brutuscat (~brutuscat@60.Red-193-152-185.dynamicIP.rima-tde.net) Quit (Remote host closed the connection)
[15:07] * overclk (~overclk@59.93.64.182) Quit (Remote host closed the connection)
[15:08] * Peaced (~Esge@tor-exit-node-nibbana.dson.org) has joined #ceph
[15:11] * TMM (~hp@178-84-46-106.dynamic.upc.nl) Quit (Remote host closed the connection)
[15:16] * TMM (~hp@178-84-46-106.dynamic.upc.nl) has joined #ceph
[15:20] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[15:20] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[15:21] <Amto_res> Hello, is it normal that the command : time rbd ls -p MYPOOL -l ... average 5-10 seconds to be executed ?
[15:25] <jdillaman> Amto_res: no ??? most likely you have something degraded in your cluster
[15:26] <Amto_res> jdillaman: my cluster is OK (HEALTH_OK) .. :|
[15:27] * vbellur (~vijay@122.171.153.113) has joined #ceph
[15:28] * yanzheng (~zhyan@182.139.23.79) Quit (Quit: This computer has gone to sleep)
[15:28] <Amto_res> If I remove the option -l command is executed instantly.
[15:28] <jdillaman> Amto_res: how many images and snapshots do you have?
[15:29] * yanzheng (~zhyan@182.139.23.79) has joined #ceph
[15:30] <Amto_res> jdillaman: average 195
[15:31] * overclk (~overclk@59.93.64.182) has joined #ceph
[15:31] * tuxcraft1r (~jelle@ebony.powercraft.nl) has joined #ceph
[15:31] * tuxcrafter (~jelle@ebony.powercraft.nl) Quit (Read error: Connection reset by peer)
[15:32] <jdillaman> Amto_res: when performing a ???rbd ls -l???, each image will require several round trips to the server for each image and snapshot.
[15:33] * ira (~ira@nat-pool-bos-u.redhat.com) has joined #ceph
[15:33] <jdillaman> perhaps you have a high latency connection between one or more OSDs and your client
[15:36] * Icey (~IceyEC_@0001bbad.user.oftc.net) has joined #ceph
[15:36] * Icey (~IceyEC_@0001bbad.user.oftc.net) Quit (Remote host closed the connection)
[15:38] * Peaced (~Esge@4Z9AAASTP.tor-irc.dnsbl.oftc.net) Quit ()
[15:38] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[15:38] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[15:39] <Amto_res> Hmmm, I do not think it's a latency problem ... :(
[15:42] <jdillaman> Amto_res: you can try to re-run with the ???--debug-rbd=20??? option to see if you can spot a particular slowdown
[15:42] * calvinx (~calvin@101.100.172.246) has joined #ceph
[15:44] * Icey (~IceyEC_@0001bbad.user.oftc.net) has joined #ceph
[15:45] * TMM (~hp@178-84-46-106.dynamic.upc.nl) Quit (Remote host closed the connection)
[15:45] * mattbenjamin (~mbenjamin@aa2.linuxbox.com) has joined #ceph
[15:47] * kefu (~kefu@114.92.106.70) has joined #ceph
[15:47] <Amto_res> jdillaman: in a debug message, librbd activate cache "librbd::ImageCtx: enabling caching..." why ?
[15:47] <Amto_res> I'll watch because I have 5000 lines ...
[15:48] <jdillaman> Amto_res: librbd starts the cache when you open the image (if enabled) ??? and the image is opened when you need to stat it, retrieve its snapshots, etc
[15:49] * vata (~vata@cable-21.246.173-197.electronicbox.net) Quit (Ping timeout: 480 seconds)
[15:50] * overclk (~overclk@59.93.64.182) Quit (Remote host closed the connection)
[15:52] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[15:52] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[15:52] <Amto_res> jdillaman: according to you, this command can be used in a routine that regularly launches? (20x every 5 minutes)
[15:53] <jdillaman> ?
[15:53] <Amto_res> jdillaman: https://github.com/OpenNebula/one/blob/master/src/vmm_mad/remotes/poll_xen_kvm.rb#L551
[15:54] <Amto_res> this script is executed every minute on a set of server
[15:54] * Icey (~IceyEC_@0001bbad.user.oftc.net) Quit (Quit: Leaving)
[15:54] * Icey (~IceyEC_@0001bbad.user.oftc.net) has joined #ceph
[15:54] * Icey (~IceyEC_@0001bbad.user.oftc.net) Quit ()
[15:55] * Icey (~IceyEC_@0001bbad.user.oftc.net) has joined #ceph
[15:56] <jdillaman> Amto_res: that code looks pretty in-efficient ??? for each device it will pull a ???rbd ls -l??? and just extract the data it needs for the one image. silly
[15:57] * pabluk is now known as pabluk_
[15:57] <jdillaman> Amto_res: it should just perform a ???rbd info --format xml <image-spec>??? to extract the size
[15:57] <flaf> Hi, can someone confirm to me this: for a ceph _client_ node, the _only_ necessary configs are 1) just [mon.$id] entries in the /etc/ceph/ceph.conf file and 2) of course the keyring file(s). Correct?
[15:58] * thomnico (~thomnico@2a01:e35:8b41:120:a966:fe0a:1aa3:4334) Quit (Ping timeout: 480 seconds)
[15:58] * thomnico (~thomnico@2a01:e35:8b41:120:d436:5201:fc0e:36a9) has joined #ceph
[15:58] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[15:58] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[15:59] * bene2 (~bene@nat-pool-bos-t.redhat.com) has joined #ceph
[15:59] <Amto_res> Many Thanks jdillaman ...
[15:59] <jdillaman> Amto_res: np
[16:00] <Amto_res> this script consumes far too many IOPS :(
[16:00] * ira (~ira@nat-pool-bos-u.redhat.com) Quit (Ping timeout: 480 seconds)
[16:01] * haomaiwang (~haomaiwan@s16.ezlink.hk) Quit (Remote host closed the connection)
[16:01] * haomaiwang (~haomaiwan@s16.ezlink.hk) has joined #ceph
[16:01] * jasuarez (~jasuarez@186.Red-83-37-101.dynamicIP.rima-tde.net) Quit (Quit: WeeChat 1.2)
[16:04] * MentalRay (~MentalRay@MTRLPQ42-1176054809.sdsl.bell.ca) has joined #ceph
[16:05] * zaitcev (~zaitcev@c-50-130-189-82.hsd1.nm.comcast.net) has joined #ceph
[16:06] * ira (~ira@nat-pool-bos-t.redhat.com) has joined #ceph
[16:08] * Icey_ (~IceyEC_@pool-74-109-7-163.phlapa.fios.verizon.net) has joined #ceph
[16:08] * Icey (~IceyEC_@0001bbad.user.oftc.net) Quit (Ping timeout: 480 seconds)
[16:09] * dgurtner (~dgurtner@77.95.96.78) has joined #ceph
[16:09] * enax (~enax@hq.ezit.hu) Quit (Ping timeout: 480 seconds)
[16:10] * rwheeler (~rwheeler@nat-pool-bos-t.redhat.com) has joined #ceph
[16:10] * Icey_ (~IceyEC_@pool-74-109-7-163.phlapa.fios.verizon.net) Quit ()
[16:10] * Icey (~IceyEC_@0001bbad.user.oftc.net) has joined #ceph
[16:13] <m0zes> flaf: under [global], I think you just need a comma separated list of ips in the mon_host entry.
[16:13] <m0zes> and a keyring.
[16:14] <m0zes> so, 2 lines.
[16:14] <m0zes> for me, at least, rbd commands are still working with just those 2 lines and a keyring.
[16:15] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[16:15] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[16:17] * tuxcraft1r is now known as tuxcrafter
[16:17] * tenshi (~David@MTRLPQ42-1176054809.sdsl.bell.ca) Quit (Read error: Connection reset by peer)
[16:21] <Be-El> radosgw newbie question: do i need any special setup if i want to provide several rgw instances as failover group?
[16:21] <Be-El> or is something like round robin dns sufficient?
[16:23] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[16:23] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[16:25] * debian112 (~bcolbert@24.126.201.64) has joined #ceph
[16:26] <Amto_res> jdillaman: it's better to make 1 rbd ls to 20 VM or 20 rbd info?
[16:28] * tenshi (~David@MTRLPQ42-1176054809.sdsl.bell.ca) has joined #ceph
[16:28] <jdillaman> Amto_res: depends on the end-goal. if OpenNebula is trying to retrieve the disk sizes for all RBD images, it should probably just cache the result of a single ???rbd ls -l??? call instead of invoking it for each and every image
[16:29] <jdillaman> Amto_res: the most efficient would be to just use ceph-ruby directly
[16:31] <flaf> m0zes: or just the [mon.$id], I think it's the same. Thx.
[16:31] <babilen> What are your opinions regarding 40GbE on ceph-public / ceph-replication in OSDs, MONs and clients? How much network traffic do MONs have to cope with?
[16:31] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[16:31] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[16:32] <babilen> ... and how many MON nodes do I need per OSD node (if the latter has ~18 OSDs each) ?
[16:32] <babilen> I plan to get 5 MON nodes anyway, but when would I have to add additional ones?
[16:33] <m0zes> babilen: 3 or 5 mon nodes. more doesn't usually help, less and you have SPOF.
[16:34] <babilen> How many OSD nodes are fine with that? I heard something like 15-20 OSD-nodes per MON node and try to get a feeling for that
[16:34] <m0zes> I use 40GbE for everything. mons don't need that much traffic, but I'm buying 40Gb switches anyway, might as well.
[16:34] <m0zes> there isn't a ratio of mons to osds.
[16:35] <babilen> Ah
[16:35] <m0zes> all mons should be handling all everything.
[16:35] <babilen> So you would put 40GbE links in MONs too and each MON needs to be able to handle everything?
[16:36] <mistur> Hello
[16:36] <m0zes> they have to agree on cluster state. which is why you want an odd number above 1. if you have more than 5, you're usually just replicating data that is already safe, and adding latency to the transactions, as they all have to be in sync to continue.
[16:36] * yanzheng (~zhyan@182.139.23.79) Quit (Quit: This computer has gone to sleep)
[16:37] <babilen> How beefy do they have to be?
[16:38] <mistur> I just installed ceph on 5 nodes (HDD 10 x 6TB , CPU 24c/48t, Memory 128GB, 1 x 10Gb ) on ubuntu 14.04 with ansible-ceph
[16:38] <mistur> I'm doing some test with rbd
[16:38] <mistur> I create a 10TB block device
[16:39] <mistur> I did a dd of 1TB file a more than 400MBytes/s
[16:39] * sankarshan (~sankarsha@121.244.87.117) has joined #ceph
[16:39] * moore (~moore@64.202.160.88) has joined #ceph
[16:39] <mistur> but read is really slow ~60MBytes/s
[16:40] <m0zes> something quad-core, 8-16GB of mem, and a good 120GB+ ssd should be sufficient for just about anything a mon should need.
[16:40] <m0zes> might even be over provisioned, but overprovisioned is better than under provisioned.
[16:40] * sankarshan (~sankarsha@121.244.87.117) Quit ()
[16:41] <babilen> m0zes: http://paste.debian.net/322188/ ??? five of them are really enough for a "larger" ceph cluster? Sounds a little "overprovisioned", but we wouldn't need anything beefier?
[16:42] <jdillaman> mistur: 400MB w/ oflag=direct?
[16:42] * yanzheng (~zhyan@182.139.23.79) has joined #ceph
[16:43] * ibravo (~ibravo@72.198.142.104) has joined #ceph
[16:43] <mistur> jdillaman: no option on dd
[16:43] <mistur> jdillaman: let me try
[16:44] <jdillaman> mistur: k ??? w/o the oflag=direct on your rbd volume you are just writing to the pagecache
[16:44] * musha68k (~zomg@h081217234161.dyn.cm.kabsi.at) has joined #ceph
[16:44] * sankarshan (~sankarsha@121.244.87.117) has joined #ceph
[16:45] * enax (~enax@94-21-125-253.pool.digikabel.hu) has joined #ceph
[16:45] <flaf> Hi, it's a little bit out of topic (sorry for that), but do you know a linux command to pretty print disks with partitions and partlabels (it's GPT)? Something like lsblk could be just _perfect_ but unfortunately this command doesn't print partlabel, just fs label (and lot of things)..
[16:46] <mistur> jdillaman: ok so, 30MB/s with oflag=direct
[16:46] <musha68k> I???m using the S3 interface with public-write-read ACL and a corresponding presigned url and I am still getting this when rgw logging is set to 20: ???Searching permissions for uid=anonymous mask=50??? ???Permissions for user not found???
[16:46] <mistur> seem to be really really slow
[16:47] <m0zes> babilen: my monitors (5 of them) even when backfilling 650TB at 2GB/s and running cluster loads were fine on less. 2PB raw space, 432 OSDs, 4 pools, 250M objects. They sat using 50-75% of a core, 2-3GB of memory. A decent number of IOPs to the ssd, but more than capable.
[16:47] <m0zes> how large are you planning on making this?
[16:47] * tuxcrafter (~jelle@ebony.powercraft.nl) Quit (Ping timeout: 480 seconds)
[16:47] <mistur> I did my first ceph install today, so I have lot's of stuff to learn about it
[16:49] <mistur> each disk can read at +200Mo/s (sequencial), I have 50 disks, I have 10Gbits/s nic on all nodes
[16:49] <m0zes> mistur: have you tried changing the block size with dd? the default object size is 4MB, and the default dd block size is 4K, leading to 1024 round trip writes per object, and would lead to some slowness.
[16:49] <mistur> I use bs=1M
[16:50] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[16:50] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[16:50] <mistur> m0zes: with bs=4M I got ~55MB/s
[16:51] * overclk (~overclk@59.93.64.182) has joined #ceph
[16:52] <m0zes> ceph really shines on parallel access, not necessarily single client performance... are you using any ssd journaling?
[16:53] <mistur> m0zes: not yet
[16:53] * Venturi (Venturi@93-103-91-169.dynamic.t-2.net) has joined #ceph
[16:53] <mistur> I have 2 SSD (raid1) for OS
[16:53] <babilen> m0zes: 200TB of storage would be fine for the beginning
[16:54] <mistur> but I have some spare space
[16:54] <mistur> I might create a partition of 200GB per node
[16:55] <mistur> but I don't know how I can define the size of the journal partition
[16:55] <m0zes> babilen: so, probably greatly overprovisioning the mons, at least for the initial buildout.
[16:55] <babilen> Well, I'd rather have too much performance than too little
[16:56] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[16:56] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[16:56] <tenshi> hi everyone
[16:56] <tenshi> version 0.94.5 released on C7 ?
[16:56] <m0zes> mistur: http://docs.ceph.com/docs/master/rados/configuration/osd-config-ref/#journal-settings
[16:56] <babilen> m0zes: Thank you *so much* for the comments. I had hoped that I get a good feeling for these numbers by playing on VMs, but that really wasn't the case. And reading/watching talks can only do so much.
[16:56] <m0zes> anyway, I wouldn't journal on any SSD unless you've tested it thoroughly
[16:57] <m0zes> http://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/
[16:57] * dgurtner (~dgurtner@77.95.96.78) Quit (Ping timeout: 480 seconds)
[16:57] <babilen> m0zes: Do you use any caching tiers ?
[16:58] <Venturi> are ceph channel logs available online?
[16:58] <m0zes> yes, I've got a cache tier for my cephfs. I use a cache tier in front of a erasure coded pool.
[16:58] <Venturi> seems like link does not work: http://irclogs.ceph.widodh.nl
[16:58] <babilen> m0zes: So it's worth looking into that?
[16:59] * overclk (~overclk@59.93.64.182) Quit (Ping timeout: 480 seconds)
[16:59] * kefu (~kefu@114.92.106.70) Quit (Max SendQ exceeded)
[16:59] <babilen> I seem to remember that sebastien wrote somewhere that it really didn't pay off / cause other problems
[17:00] <m0zes> babilen: if you're doing erasure coded pools and either RBD or cephfs with them, you have to. at their current stage, they work well enough, but there are definitely performance tuning that needs to happen in the code.
[17:00] <mistur> m0zes: even if on 2 SSD DC in raid1 hw ?
[17:01] <babilen> I have not yet decided if EC will be used or not
[17:01] * haomaiwang (~haomaiwan@s16.ezlink.hk) Quit (Remote host closed the connection)
[17:01] * haomaiwang (~haomaiwan@s16.ezlink.hk) has joined #ceph
[17:01] <tenshi> I just upgraded to latest version of Hammer, I get ceph version 0.94.5, is that normal ?
[17:01] * lookcrabs (~lookcrabs@tail.seeee.us) has joined #ceph
[17:01] <tenshi> shouldn t be 0.94.4 ?
[17:02] <lincolnb> 0.94.5 is out.
[17:02] <lincolnb> as of 10/26
[17:02] <lincolnb> fixes a critical regression in librbd. http://docs.ceph.com/docs/master/_downloads/v0.94.5.txt for changelog
[17:02] * haomaiwang (~haomaiwan@s16.ezlink.hk) Quit ()
[17:03] <tenshi> thank you, thought it was 0.94.4 last due to the blog: http://ceph.com/community/blog/
[17:03] <lincolnb> ahh
[17:03] * ircolle (~Adium@2601:285:201:2bf9:11d6:1c24:cfdb:572d) has joined #ceph
[17:03] <m0zes> mistur: which ones? and in general, I avoid raid'ing SSDs from the same manufacturer. they tend to wear at the same rate and fail at the same time.
[17:03] <babilen> m0zes: Which NICs and switch(es) do you guys use?
[17:04] * Docta (~Docta@pool-108-46-37-8.nycmny.fios.verizon.net) Quit (Ping timeout: 480 seconds)
[17:04] <tenshi> thkx lincolnb
[17:04] <mistur> m0zes: Intel DC S3500
[17:05] * kawa2014 (~kawa@93-62-73-252.ip21.fastwebnet.it) Quit (Ping timeout: 480 seconds)
[17:07] <m0zes> babilen: Mellanox Connect-X 3 NICs, they perform pretty well, support ROCE, and can be put in IB mode in the event we need to switch someday (we put the same cards in our compute nodes). as for switches http://www.penguincomputing.com/products/network-switches/arctica-3200xl/
[17:07] * analbeard (~shw@support.memset.com) Quit (Quit: Leaving.)
[17:09] * calvinx (~calvin@101.100.172.246) Quit (Quit: calvinx)
[17:09] <m0zes> mistur: the 3500 series aren't rated for a *lot* of writes, and every write to ceph has to hit the journal for every osd it goes to.
[17:09] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[17:09] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[17:09] <m0zes> I've heard people do it...
[17:10] <mistur> m0zes: in the futur I will have 25 nodes in my ceph cluster (SSD 240GB, HDD 10 x 6TB , CPU 24c/48t, Memory 128GB, 1 x 10Gbs/s)
[17:10] * vata (~vata@207.96.182.162) has joined #ceph
[17:11] <mistur> so it's one journal per osds, so if I decide to use a partition on SSD, the size should be at least numberof(OSD) * sizeof(journal)
[17:12] <m0zes> mistur: anyway, performance, the spinning disks probably hit 110-130MB/s sequential writes, cut that in half, cause you're journaling and writing to the same disk. you've hit about that with your writes to RBD, using 1-4MB block sizes.
[17:13] <m0zes> mistur: ideally, it would be 1 partition per osd journal.
[17:13] <m0zes> let ceph use the raw partition for the journal
[17:13] <mistur> In my test, I got 200MB/s
[17:13] * davidzlap (~Adium@cpe-172-91-154-245.socal.res.rr.com) Quit (Read error: Connection reset by peer)
[17:13] <m0zes> you said reads above.
[17:13] * ibravo (~ibravo@72.198.142.104) Quit (Quit: Leaving)
[17:13] * davidzlap (~Adium@2605:e000:1313:8003:d491:863a:7c6e:2f4a) has joined #ceph
[17:13] <mistur> I mean single disk seq write test
[17:14] <m0zes> and were you using oflag=direct when testing the disks themselves?
[17:15] * squizzi (~squizzi@nat-pool-rdu-u.redhat.com) has joined #ceph
[17:15] * haomaiwang (~haomaiwan@s16.ezlink.hk) has joined #ceph
[17:15] * kawa2014 (~kawa@2001:67c:1560:8007::aac:c1a6) has joined #ceph
[17:15] <mistur> I did some sequentiel write test on disks before install ceph and I got ~200MB/s write speed per disk
[17:15] <MentalRay> mistur, which drive model you had 200MB/s?
[17:17] <mistur> MentalRay: Ultrastar 7K6000 : https://www.hgst.com/products/hard-drives/ultrastar-7k6000
[17:18] <MentalRay> let me get you the command to test direct
[17:18] <MentalRay> like m0zes mention
[17:19] <mistur> m0zes: my OS is install on the SSD, how I can say to ceph to use the rest of partition ?
[17:19] * TMM (~hp@178-84-46-106.dynamic.upc.nl) has joined #ceph
[17:19] <mistur> s/partition/disk/
[17:20] * yanzheng (~zhyan@182.139.23.79) Quit (Quit: This computer has gone to sleep)
[17:21] * qhartman (~qhartman@den.direwolfdigital.com) Quit (Quit: Ex-Chat)
[17:21] <m0zes> since your ssd is already in use, you'd need to prepare the journal partitions manually. once that is done, you can flush the osd journal that is on the spinners, change journal to symlink to a uniq partition per osd, and start the osd.
[17:22] <m0zes> fairly in depth, which one of the reasons why ceph recommends dedicated journal ssds.
[17:22] <mistur> m0zes: I'm usinf ansible-ceph, I gonna check how I can do that tomorow
[17:23] * jwilkins (~jowilkin@67.204.149.211) has joined #ceph
[17:23] <mistur> here it's 5:23pm
[17:24] * squizzi (~squizzi@nat-pool-rdu-u.redhat.com) Quit (Ping timeout: 480 seconds)
[17:24] <mistur> MentalRay: I'll do to test with oflag=direct on disk as well
[17:24] <mistur> MentalRay: I'll do some tests with oflag=direct on disk as well *
[17:25] * doppelgrau (~doppelgra@pd956d116.dip0.t-ipconnect.de) has joined #ceph
[17:26] <mistur> m0zes, MentalRay: thanks for your help
[17:26] <m0zes> no problem. community and all that :D
[17:27] * joshd (~jdurgin@68-119-140-18.dhcp.ahvl.nc.charter.com) has joined #ceph
[17:27] <mistur> :)
[17:28] * overclk (~overclk@59.93.64.182) has joined #ceph
[17:29] <MentalRay> I miss the beginning but I only have one recommendation
[17:29] <MentalRay> take the time to test SSD for your journal
[17:29] <MentalRay> also if there is one place you dont want to try to save money its on the journal as well
[17:29] <MentalRay> We run a full SSD cluster and the bottleneck for us is the journals
[17:30] * kawa2014 (~kawa@2001:67c:1560:8007::aac:c1a6) Quit (Ping timeout: 480 seconds)
[17:30] <mistur> MentalRay: it's a POC
[17:30] <m0zes> in a full ssd cluster, wouldn't it be prudent to be journaling on the same ssd that the osd lives?
[17:31] <MentalRay> it could but the ssd we use for storage tierd are not great for direct writes
[17:31] <MentalRay> so we externalize the journal for that
[17:32] <mistur> so it might be safer to keep journal on hdd if ssd are not enouph fast/trustable ?
[17:32] <MentalRay> we got good result with intel dc sad
[17:32] <mistur> MentalRay: ok
[17:33] <MentalRay> you planning to use what ssd during your POC?
[17:33] <mistur> and at least, I will have 25 nodes in the cluster, so in case of ssd crash on a node, I gonna loose 1/25 of replica
[17:34] <mistur> MentalRay: DC S3500
[17:34] <MentalRay> how many osd per node?
[17:34] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[17:34] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[17:34] * TheSov (~thesov@64.241.45.209) has joined #ceph
[17:35] * squizzi (~squizzi@nat-pool-rdu-t.redhat.com) has joined #ceph
[17:35] <mistur> MentalRay: the 25 nodes are all the identical : SSD 240GB DC S3500, HDD 10 x 6TB Ultrastar 7K6000, CPU E5-2680 24c/48t 128GB, 1 x 10Gbs/s
[17:35] <mistur> MentalRay: so 10 osd per node
[17:35] <MentalRay> put more than 1 ssd per node
[17:36] <mistur> it 2x SSD 240GB DC S3500
[17:36] <mistur> in raid1 hw
[17:37] <MentalRay> I would put at least 2 journal per node
[17:37] <MentalRay> base on the hw you gonna use
[17:37] <MentalRay> its only a few hundreds more per node
[17:37] <MentalRay> $
[17:37] <mistur> you mean add more ssd ?
[17:37] <MentalRay> yes
[17:38] <MentalRay> everything has to go thru the journals
[17:38] <MentalRay> (writes)
[17:38] <MentalRay> doing 1 ssd for 10hdd is low in my opinion
[17:38] * `10` (~10@69.169.91.14) Quit (Ping timeout: 480 seconds)
[17:39] <musha68k> I???m fighting with erratic behaviour on my https://github.com/ceph/ceph-ansible supported local instance of rgw - could somebody help me out on a few simple questions with regards to rgw/s3?
[17:39] <mistur> We ask for that but seem to not be an option right now, we don't have more space in the server, we have to add extra disk bay on the back with extra raid card etc...
[17:39] <mistur> it's not cheap
[17:39] * cholcombe (~chris@c-73-180-29-35.hsd1.or.comcast.net) has joined #ceph
[17:40] <mistur> but I keep the argument
[17:40] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[17:40] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[17:40] <mistur> might be uselful for the futur
[17:40] <musha68k> I???m developing an application which needs to talk to ceph via the s3 interface and strangely enough the ???anonymous??? user worked for me but not on every setup of a fresh ceph cluster instance.
[17:40] <MentalRay> this environment will be write intensive?
[17:40] <mistur> MentalRay: not so much, it's to host public dataset for research
[17:41] <mistur> so most of the time it will be read, write will be only to sync dataset
[17:41] <musha68k> The thing is that I???m using multipart upload parts, generating presigned urls for them so that a browser client can PUT to them without us having to set up a ceph user (it should work through the generated signature alone).
[17:42] <MentalRay> ok ok
[17:42] <MentalRay> other questions:
[17:42] <musha68k> Do I actually have to set-up an ???anonymous??? user for it to work? The docs say this is actually implicit.
[17:42] <musha68k> IIIR
[17:42] <musha68k> C
[17:42] <MentalRay> 1x 10G per node? meaning your internal ceph network and storage network will be on the same card?
[17:43] * olid15 (~olid1982@aftr-185-17-204-236.dynamic.mnet-online.de) has joined #ceph
[17:43] <musha68k> :(
[17:43] <mistur> MentalRay: for now yes
[17:43] * lpabon (~quassel@24-151-54-34.dhcp.nwtn.ct.charter.com) Quit (Ping timeout: 480 seconds)
[17:43] <mistur> we might plug the second 10G nic in the futur
[17:45] <flaf> I have installed a Infernalis test cluster. All is fine (Health ok) but with ???ceph -s??? I can see ???flags sortbitwise???. Personally I haven't set this flag. Do you know its meaning?
[17:45] <mistur> but due to it's a POC, I we need more, that's mean, it's not a POC anymore and we will have to invest in a bigger cluster
[17:46] * sudocat (~dibarra@192.185.1.20) has joined #ceph
[17:48] * drankis (~drankis__@89.111.13.198) Quit (Ping timeout: 480 seconds)
[17:48] * overclk (~overclk@59.93.64.182) Quit (Quit: Leaving...)
[17:49] * olid14 (~olid1982@aftr-185-17-204-236.dynamic.mnet-online.de) Quit (Ping timeout: 480 seconds)
[17:49] * wschulze (~wschulze@cpe-72-225-192-123.nyc.res.rr.com) has joined #ceph
[17:53] <m0zes> flaf: there is a *little* bit of background here https://github.com/ceph/ceph/pull/5470
[17:54] <flaf> ah thx for the link m0zes. ;) I'm reading.
[17:54] * sankarshan (~sankarsha@121.244.87.117) Quit (Quit: Are you sure you want to quit this channel (Cancel/Ok) ?)
[17:55] <musha68k> Do I have to set-up an ???anonymous??? rgw user on a fresh install?
[17:55] <musha68k> The documentation is ambigous about this.
[17:56] * ktdreyer (~kdreyer@polyp.adiemus.org) Quit (Ping timeout: 480 seconds)
[17:57] * b0e (~aledermue@213.95.25.82) Quit (Quit: Leaving.)
[17:58] * shylesh (~shylesh@59.95.68.167) has joined #ceph
[17:59] * drankis (~drankis__@46.109.81.218) has joined #ceph
[18:00] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[18:00] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[18:00] * dugravot6 (~dugravot6@dn-infra-04.lionnois.univ-lorraine.fr) Quit (Quit: Leaving.)
[18:01] * haomaiwang (~haomaiwan@s16.ezlink.hk) Quit (Remote host closed the connection)
[18:01] * haomaiwang (~haomaiwan@s16.ezlink.hk) has joined #ceph
[18:03] * dgurtner (~dgurtner@185.10.235.250) has joined #ceph
[18:06] * ktdreyer (~kdreyer@polyp.adiemus.org) has joined #ceph
[18:07] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[18:07] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[18:07] * derjohn_mob (~aj@fw.gkh-setu.de) Quit (Ping timeout: 480 seconds)
[18:08] <musha68k> leseb_: could you help me out on why I can???t upload to a presigned s3 multipart upload part (created via my ???testuser???) via PUT (from ???anonymous???)? I???m using your ansible/ceph setup - it???s working flawlessly, mostly..
[18:11] * ksperis (~laurent@46.218.42.103) Quit (Quit: Leaving.)
[18:11] * kawa2014 (~kawa@2001:67c:1560:8007::aac:c1a6) has joined #ceph
[18:14] * lpabon (~quassel@pool-72-74-184-69.bstnma.east.verizon.net) has joined #ceph
[18:16] * nardial (~ls@ipservice-092-209-179-024.092.209.pools.vodafone-ip.de) has joined #ceph
[18:17] <musha68k> Strangely enough I think at one point I had it running already, which steps do I need to go through in order to make public-write ACL???d anonymous PUT requests on presigned multipart upload parts available?
[18:20] * pabluk_ is now known as pabluk
[18:21] * derjohn_mob (~aj@88.128.80.100) has joined #ceph
[18:21] * jdillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) Quit (Quit: jdillaman)
[18:22] * TMM (~hp@178-84-46-106.dynamic.upc.nl) Quit (Quit: Ex-Chat)
[18:25] * shaunm (~shaunm@208.102.161.229) Quit (Ping timeout: 480 seconds)
[18:26] * Kupo1 (~tyler.wil@23.111.254.159) has joined #ceph
[18:26] <MentalRay> mistur, ok :)
[18:26] <MentalRay> if you can keep me posted
[18:26] * mattch (~mattch@pcw3047.see.ed.ac.uk) Quit (Quit: Leaving.)
[18:26] * mattch (~mattch@pcw3047.see.ed.ac.uk) has joined #ceph
[18:27] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[18:27] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[18:27] * xarses (~xarses@118.103.8.153) Quit (Remote host closed the connection)
[18:30] * thesix (~thesix@leifhelm.hofos.at) Quit (Remote host closed the connection)
[18:30] * thesix (~thesix@leifhelm.hofos.at) has joined #ceph
[18:31] * kawa2014 (~kawa@2001:67c:1560:8007::aac:c1a6) Quit (Ping timeout: 480 seconds)
[18:31] * enax (~enax@94-21-125-253.pool.digikabel.hu) Quit (Ping timeout: 480 seconds)
[18:31] * jdillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) has joined #ceph
[18:32] * xarses (~xarses@118.103.8.153) has joined #ceph
[18:32] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[18:32] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[18:34] * ade (~abradshaw@tmo-111-182.customers.d1-online.com) Quit (Ping timeout: 480 seconds)
[18:35] * lpabon (~quassel@pool-72-74-184-69.bstnma.east.verizon.net) Quit (Ping timeout: 480 seconds)
[18:37] * Icey (~IceyEC_@0001bbad.user.oftc.net) Quit (Ping timeout: 480 seconds)
[18:38] * tdasilva (~tdasilva@nat-pool-bos-t.redhat.com) has joined #ceph
[18:38] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[18:38] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[18:40] * garphy is now known as garphy`aw
[18:40] * kutija (~kutija@95.180.90.38) Quit (Max SendQ exceeded)
[18:41] * dgurtner_ (~dgurtner@185.10.235.250) has joined #ceph
[18:41] * kutija (~kutija@95.180.90.38) has joined #ceph
[18:43] * dgurtner (~dgurtner@185.10.235.250) Quit (Ping timeout: 480 seconds)
[18:43] * thomnico (~thomnico@2a01:e35:8b41:120:d436:5201:fc0e:36a9) Quit (Quit: Ex-Chat)
[18:46] * mattronix (~quassel@2a02:898:162:1::1) Quit (Remote host closed the connection)
[18:47] * alfredodeza (~alfredode@198.206.133.89) Quit (Quit: ZNC - http://znc.in)
[18:47] * alfredodeza (~alfredode@198.206.133.89) has joined #ceph
[18:50] * thomnico (~thomnico@2a01:e35:8b41:120:6d41:9276:37c5:3038) has joined #ceph
[18:50] * pabluk is now known as pabluk_
[18:54] * rwheeler (~rwheeler@nat-pool-bos-t.redhat.com) Quit (Quit: Leaving)
[18:54] * shaunm (~shaunm@172.56.28.239) has joined #ceph
[18:58] * Icey (~IceyEC_@0001bbad.user.oftc.net) has joined #ceph
[18:58] * madkiss (~madkiss@2001:6f8:12c3:f00f:195b:ce41:fb81:28e7) has joined #ceph
[18:59] * Sami345 (~Sketchfil@nl9x.mullvad.net) has joined #ceph
[19:01] * haomaiwang (~haomaiwan@s16.ezlink.hk) Quit (Remote host closed the connection)
[19:01] * haomaiwang (~haomaiwan@s16.ezlink.hk) has joined #ceph
[19:01] * ngoswami (~ngoswami@121.244.87.116) Quit (Quit: Leaving)
[19:01] * yanzheng (~zhyan@182.139.23.79) has joined #ceph
[19:01] * ron-slc (~Ron@173-165-129-118-utah.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[19:03] * ron-slc (~Ron@173-165-129-118-utah.hfc.comcastbusiness.net) has joined #ceph
[19:03] * `10` (~10@69.169.91.14) has joined #ceph
[19:06] * bara (~bara@213.175.37.10) Quit (Quit: Bye guys! (??????????????????? ?????????)
[19:11] * rakeshgm (~rakesh@106.51.30.186) has joined #ceph
[19:13] * jonfatino (~jonfatino@107.181.176.252) has joined #ceph
[19:14] * jordanP (~jordan@204.13-14-84.ripe.coltfrance.com) Quit (Quit: Leaving)
[19:14] <jonfatino> Hi everyone. I am looking to build a cdn system with about 10 servers around the world in different locations. Each server would need to run ceph + nginx to serve static files and have very little writes going on. Mostly reads for nginx. Anyone have a good guide or has setup something similar to this?
[19:15] * yanzheng (~zhyan@182.139.23.79) Quit (Quit: This computer has gone to sleep)
[19:15] * xarses (~xarses@118.103.8.153) Quit (Remote host closed the connection)
[19:15] <jonfatino> The servers have dual quad core procs and lots of ram and fast enterprise disks. They also have dedicated 1Gbps ports. So hardware shouldn't be an issue.
[19:15] * DV (~veillard@2001:41d0:1:d478::1) Quit (Ping timeout: 480 seconds)
[19:16] * krypto (~krypto@103.252.27.248) has joined #ceph
[19:16] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[19:16] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Remote host closed the connection)
[19:16] <jonfatino> Some system where ceph can replicate the same block between all 10 servers and cephfs (fuse) connect to that block locally to serve nginx content files would be ideal. Anyone have a guide?
[19:17] * thomnico (~thomnico@2a01:e35:8b41:120:6d41:9276:37c5:3038) Quit (Quit: Ex-Chat)
[19:18] <itwasntandy> unning cephfs on the same systems as the cluster is not recommended according tot he FAQ, as can cause memory starvation and a deadlock
[19:19] * xarses (~xarses@118.103.8.153) has joined #ceph
[19:19] <jonfatino> itwasntandy: I could install linux kvm and create two vms and run ceph on one and then nginx on another and just have it mount cephfs from the other server
[19:20] <itwasntandy> oh sure
[19:20] <itwasntandy> you could just run the object gateway too
[19:21] <jonfatino> I current am experimenting with glusterfs and it is working great however I am looking at alternative options just to see what is out there. Gluster brick is on each server running glusterd and I connect to that local brick using glusterfs and serve nginx content
[19:21] <itwasntandy> and serve from that via apache/nginx
[19:21] * mykola (~Mikolaj@91.225.203.72) has joined #ceph
[19:21] <erhudy> i would probably serve from RADOSgw and munge the paths in nginx/apache in whatever way necessary
[19:22] * xarses (~xarses@118.103.8.153) Quit (Remote host closed the connection)
[19:22] * Be-El (~quassel@fb08-bcf-pc01.computational.bio.uni-giessen.de) Quit (Remote host closed the connection)
[19:22] <jonfatino> Doing a object storage call for each file load is that going to take a bit? This client has 3+ million very small files. (mp3's etc)
[19:23] <jonfatino> Would probably be easier using the client (fuse) and mount /var/www for nginx rather than some complex object api call every time want to load a file
[19:23] <erhudy> well, it's not the kind of thing that blanket statements can be made about, you'd need to do your own performance testing based on your use case
[19:23] <erhudy> it's not an API call, RADOSgw is S3
[19:23] <erhudy> just an HTTP GET
[19:25] * branto1 (~branto@ip-213-220-232-132.net.upcbroadband.cz) Quit (Quit: Leaving.)
[19:25] <jonfatino> Interesting. Do you know of any type of ceph cache? I had the idea of having a centeral storage cluster and every distro server just call that file from the cluster
[19:25] <jonfatino> The oringal first load would be a little slow but after that file is in cache it would be a lot faster.
[19:26] <erhudy> well, there's this thing: http://docs.ceph.com/docs/v0.80/radosgw/federated-config/
[19:26] <erhudy> i think there is much greater sophistication in terms of georeplication coming down the pipe
[19:26] * xarses (~xarses@118.103.8.153) has joined #ceph
[19:27] * dugravot6 (~dugravot6@2a01:e35:8bbf:4060:bc92:de00:ba1e:800d) has joined #ceph
[19:27] * dugravot6 (~dugravot6@2a01:e35:8bbf:4060:bc92:de00:ba1e:800d) Quit ()
[19:28] * krypto (~krypto@103.252.27.248) Quit (Quit: Leaving)
[19:28] <jonfatino> Interesting.
[19:28] * dgurtner (~dgurtner@185.10.235.250) has joined #ceph
[19:29] <jonfatino> Anyone else have any ideas?
[19:29] <erhudy> whole lot of rsync :v
[19:29] <jonfatino> The oringal setup was rsync and that can be a mess. I was able to clean it up a bit using lsyncd however need something that can scale outwards
[19:29] * Sami345 (~Sketchfil@5P6AAATLA.tor-irc.dnsbl.oftc.net) Quit ()
[19:29] <jonfatino> Its only about 1TB of data and 3 million small files
[19:29] * ira (~ira@nat-pool-bos-t.redhat.com) Quit (Ping timeout: 480 seconds)
[19:30] * dgurtner_ (~dgurtner@185.10.235.250) Quit (Ping timeout: 480 seconds)
[19:30] * rakeshgm (~rakesh@106.51.30.186) Quit (Quit: Leaving)
[19:31] * TMM (~hp@178-84-46-106.dynamic.upc.nl) has joined #ceph
[19:35] * shaunm (~shaunm@172.56.28.239) Quit (Ping timeout: 480 seconds)
[19:35] * thomnico (~thomnico@2a01:e35:8b41:120:6d41:9276:37c5:3038) has joined #ceph
[19:36] * dupont-y (~dupont-y@familledupont.org) has joined #ceph
[19:36] * iPrime (~oftc-webi@host217-114-156-249.pppoe.mark-itt.net) has joined #ceph
[19:37] * stiopa (~stiopa@cpc73828-dals21-2-0-cust630.20-2.cable.virginm.net) has joined #ceph
[19:38] * Diablothein (~notarima@ns213861.ovh.net) has joined #ceph
[19:40] <Venturi> would a technology such as drbd9/drbd8 be a good choice to replicate rados pool between two ceph clusters which are on two geo locations?
[19:41] * ira (~ira@nat-pool-bos-u.redhat.com) has joined #ceph
[19:41] * dyasny (~dyasny@198.251.57.99) Quit (Ping timeout: 480 seconds)
[19:43] <Gugge-47527> Venturi: what about a zpool on each location, with zfs send/receive from the master on changes?
[19:43] * ira_ (~ira@nat-pool-bos-t.redhat.com) has joined #ceph
[19:43] * ira_ (~ira@nat-pool-bos-t.redhat.com) Quit (Remote host closed the connection)
[19:43] * ira_ (~ira@nat-pool-bos-t.redhat.com) has joined #ceph
[19:45] <Venturi> possible. would zfs be than used as a filesystem of choice instead of XFS?
[19:46] <Venturi> do you have some examples of architecture with ZFS and CEPH?
[19:46] * derjohn_mob (~aj@88.128.80.100) Quit (Ping timeout: 480 seconds)
[19:47] <TheSov> Venturi, since ZOL is utter shit, and there is no port of ceph on bsd. no
[19:48] <TheSov> or rather not at this time
[19:48] * shaunm (~shaunm@172.56.28.239) has joined #ceph
[19:48] <TheSov> im sure in the future there will be a ceph port on bsd
[19:49] <TheSov> in fact im counting on IX to do just that; and include ceph into freenas
[19:51] * ira (~ira@nat-pool-bos-u.redhat.com) Quit (Ping timeout: 480 seconds)
[19:55] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[19:55] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[19:57] <Venturi> TheSov, so if i understand you, drbd is still a good choice? :)
[19:58] <TheSov> Venturi, i didnt say resort to archaic technologoies
[19:58] <jonfatino> DRDB still in production these days? I thought most people were moving towards the Corosync and pacemaker stuff
[19:58] <jonfatino> aka -> http://clusterlabs.org/
[19:59] <TheSov> u can use crush to do exactly what you are talking about and replicate an RBD pool to multiple locations
[20:00] <TheSov> Venturi, what you are asking is built into ceph, would you use a stone adz to fell a tree today? no you would use a sawzall or something similar because an adz is barbaric
[20:01] * haomaiwang (~haomaiwan@s16.ezlink.hk) Quit (Remote host closed the connection)
[20:01] * haomaiwang (~haomaiwan@s16.ezlink.hk) has joined #ceph
[20:02] <zao_> jonfatino: Oh yes.
[20:02] <zao_> Don't touch a working-enough stable-and-boring system :)
[20:02] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[20:02] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[20:03] <jonfatino> :-) I used to run DRDB. But I recently switched to linux ha and corosync and pacemaker
[20:03] * drankis (~drankis__@46.109.81.218) Quit (Quit: Leaving)
[20:04] <TheSov> jonfatino, thats awesome
[20:04] * thomnico (~thomnico@2a01:e35:8b41:120:6d41:9276:37c5:3038) Quit (Ping timeout: 480 seconds)
[20:04] <TheSov> im still trying to figure out if exporting cephfs via nfs is stable
[20:05] <TheSov> so far i havent had any issues
[20:05] <TheSov> but in theory you can mount cephfs on multiple systems
[20:05] <TheSov> and the MDS will co-ordinate locks and such
[20:05] <TheSov> also with nfs 4.1 you can build HA nfs shares
[20:05] <TheSov> with multipathing!
[20:06] * bene3 (~bene@nat-pool-bos-t.redhat.com) has joined #ceph
[20:06] * bene2 (~bene@nat-pool-bos-t.redhat.com) Quit (Ping timeout: 480 seconds)
[20:08] * Diablothein (~notarima@4Z9AAAS72.tor-irc.dnsbl.oftc.net) Quit ()
[20:08] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[20:08] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[20:10] * `10` (~10@69.169.91.14) Quit (Ping timeout: 480 seconds)
[20:12] * davidzlap1 (~Adium@2605:e000:1313:8003:d491:863a:7c6e:2f4a) has joined #ceph
[20:12] * davidzlap (~Adium@2605:e000:1313:8003:d491:863a:7c6e:2f4a) Quit (Read error: Connection reset by peer)
[20:15] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[20:15] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[20:16] * segutier (~segutier@sfo-vpn1.shawnlower.net) has joined #ceph
[20:17] * `10` (~10@69.169.91.14) has joined #ceph
[20:21] * shylesh (~shylesh@59.95.68.167) Quit (Remote host closed the connection)
[20:28] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[20:28] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Remote host closed the connection)
[20:29] * nardial (~ls@ipservice-092-209-179-024.092.209.pools.vodafone-ip.de) Quit (Quit: Leaving)
[20:29] * shaunm (~shaunm@172.56.28.239) Quit (Ping timeout: 480 seconds)
[20:33] * mattronix (~quassel@server1.mattronix.nl) has joined #ceph
[20:37] * mykola (~Mikolaj@91.225.203.72) Quit (Quit: away)
[20:38] * mattronix (~quassel@server1.mattronix.nl) Quit (Read error: Network is unreachable)
[20:44] * telnoratti (~telnoratt@telnorattitoldyou.so) Quit (Ping timeout: 480 seconds)
[20:46] * telnoratti (~telnoratt@2a00:1768:1001:37:101:b00b:5:101) has joined #ceph
[20:52] * kutija (~kutija@95.180.90.38) Quit (Quit: Textual IRC Client: www.textualapp.com)
[21:00] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[21:00] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Remote host closed the connection)
[21:00] * TheSov (~thesov@64.241.45.209) Quit (Quit: Leaving)
[21:01] * haomaiwang (~haomaiwan@s16.ezlink.hk) Quit (Remote host closed the connection)
[21:01] * haomaiwang (~haomaiwan@s16.ezlink.hk) has joined #ceph
[21:08] * diq (~diq@2620:11c:f:2:c23f:d5ff:fe62:112c) has joined #ceph
[21:08] * diq_ (~diq@2620:11c:f:2:c23f:d5ff:fe62:112c) Quit (Quit: Leaving)
[21:08] * thomnico (~thomnico@2a01:e35:8b41:120:6d41:9276:37c5:3038) has joined #ceph
[21:14] * brad[] (~Brad@184-94-17-26.dedicated.allstream.net) has joined #ceph
[21:15] * rendar (~I@host177-119-dynamic.57-82-r.retail.telecomitalia.it) Quit (Ping timeout: 480 seconds)
[21:17] * rendar (~I@host177-119-dynamic.57-82-r.retail.telecomitalia.it) has joined #ceph
[21:18] * Concubidated (~Adium@pool-173-55-225-216.lsanca.fios.verizon.net) has joined #ceph
[21:21] * mattronix (~quassel@2a02:898:162:1::1) has joined #ceph
[21:22] * rwheeler (~rwheeler@nat-pool-bos-u.redhat.com) has joined #ceph
[21:23] * angdraug (~angdraug@12.164.168.117) has joined #ceph
[21:31] * dyasny (~dyasny@38.108.87.20) has joined #ceph
[21:32] * rwheeler (~rwheeler@nat-pool-bos-u.redhat.com) Quit (Quit: Leaving)
[21:33] * _28_ria (~kvirc@194.226.6.106) has joined #ceph
[21:37] * ira_ (~ira@nat-pool-bos-t.redhat.com) Quit (Ping timeout: 480 seconds)
[21:37] * mattronix (~quassel@2a02:898:162:1::1) Quit (Remote host closed the connection)
[21:39] * mattronix (~quassel@2a02:898:162:1::1) has joined #ceph
[21:39] * dgurtner_ (~dgurtner@185.10.235.250) has joined #ceph
[21:39] * thomnico (~thomnico@2a01:e35:8b41:120:6d41:9276:37c5:3038) Quit (Quit: Ex-Chat)
[21:41] * dgurtner (~dgurtner@185.10.235.250) Quit (Ping timeout: 480 seconds)
[21:41] * Destreyf (~quassel@email.newagecomputers.info) has joined #ceph
[21:49] * mattronix (~quassel@2a02:898:162:1::1) Quit (Remote host closed the connection)
[21:50] * rotbeard (~redbeard@2a02:908:df11:4280:6267:20ff:feb7:c20) Quit (Quit: Leaving)
[21:51] * mattronix (~quassel@2a02:898:162:1::1) has joined #ceph
[21:51] * dyasny (~dyasny@38.108.87.20) Quit (Ping timeout: 480 seconds)
[21:56] * huats (~quassel@stuart.objectif-libre.com) Quit (Ping timeout: 480 seconds)
[21:58] * shaunm (~shaunm@208.102.161.229) has joined #ceph
[22:00] * dupont-y (~dupont-y@familledupont.org) Quit (Remote host closed the connection)
[22:01] * haomaiwang (~haomaiwan@s16.ezlink.hk) Quit (Remote host closed the connection)
[22:01] * haomaiwang (~haomaiwan@s16.ezlink.hk) has joined #ceph
[22:07] * neurodrone_ (~neurodron@pool-100-1-89-227.nwrknj.fios.verizon.net) has joined #ceph
[22:11] * dgurtner (~dgurtner@185.10.235.250) has joined #ceph
[22:13] * dgurtner_ (~dgurtner@185.10.235.250) Quit (Ping timeout: 480 seconds)
[22:17] * olid16 (~olid1982@aftr-185-17-204-236.dynamic.mnet-online.de) has joined #ceph
[22:17] * olid15 (~olid1982@aftr-185-17-204-236.dynamic.mnet-online.de) Quit (Read error: Connection reset by peer)
[22:20] * georgem (~Adium@206.108.127.16) Quit (Quit: Leaving.)
[22:20] * robert_ (uid109993@id-109993.brockwell.irccloud.com) has joined #ceph
[22:22] * m3porter (~m3porter@2604:a580:1:0:195c:4453:38c2:ae68) has joined #ceph
[22:26] * fridim_ (~fridim@56-198-190-109.dsl.ovh.fr) Quit (Ping timeout: 480 seconds)
[22:29] <m3porter> is it possible to change erasure coding? starting off with 4 nodes with plans to move to 7. my hope was to start with k=2,m=2 and eventually go to k=5,m=2 as we add nodes.
[22:31] <m0zes> I'm pretty sure that isn't changeable
[22:36] * Icey (~IceyEC_@0001bbad.user.oftc.net) Quit (Quit: Leaving)
[22:36] * Icey (~IceyEC_@0001bbad.user.oftc.net) has joined #ceph
[22:36] * haomaiwa_ (~haomaiwan@s16.ezlink.hk) has joined #ceph
[22:37] * haomaiwang (~haomaiwan@s16.ezlink.hk) Quit (Ping timeout: 480 seconds)
[22:37] <flaf> m0zes: I have read "your" link (https://github.com/ceph/ceph/pull/5470). To be honest, that's not all clear for me. This flag (sortbitwise) seems to be something very "low level".
[22:38] <flaf> Do you have understood something concerning this flag?
[22:38] <m0zes> it is lowlevel. there was talk about hiding the flag, but it hasn't happened yet, apparently.
[22:38] <flaf> Ok, as a user of ceph, I can forget this flag and let expert thinking about this. ;)
[22:40] * m0zes is just a user as well. sortbitwise is the new default, and supposedly the math says it is better. I wouldn't worry about it.
[22:41] <flaf> :??
[22:41] <flaf> :)
[22:45] <Venturi> TheSov: This is i guess the perfered way to do geo-replication for ceph rbd images: https://www.rapide.nl/blog/item/ceph_-_rbd_replication.html
[22:49] * erice (~eric@c-76-120-53-165.hsd1.co.comcast.net) has joined #ceph
[22:50] * xarses (~xarses@118.103.8.153) Quit (Ping timeout: 480 seconds)
[22:51] * huats (~quassel@stuart.objectif-libre.com) has joined #ceph
[22:52] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[22:54] * m3porter (~m3porter@2604:a580:1:0:195c:4453:38c2:ae68) Quit (Remote host closed the connection)
[22:54] * georgem (~Adium@184.151.178.151) has joined #ceph
[22:54] * m3porter (~m3porter@2604:a580:1:0:195c:4453:38c2:ae68) has joined #ceph
[22:57] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[22:57] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[22:59] * bene3 (~bene@nat-pool-bos-t.redhat.com) Quit (Quit: Konversation terminated!)
[22:59] * huats (~quassel@stuart.objectif-libre.com) Quit (Ping timeout: 480 seconds)
[23:01] * haomaiwa_ (~haomaiwan@s16.ezlink.hk) Quit (Remote host closed the connection)
[23:01] * musha68k (~zomg@h081217234161.dyn.cm.kabsi.at) Quit (Ping timeout: 480 seconds)
[23:01] * haomaiwang (~haomaiwan@45.32.13.54) has joined #ceph
[23:03] * tdasilva (~tdasilva@nat-pool-bos-t.redhat.com) Quit (Quit: Leaving)
[23:08] <mfa298_> jonfatino: (reading scrollback) I'm not sure that ceph is really a good fit for sharing data around like that - reads for an object come from the primary for that placement group which could be on the opposite side of the world. Also cephfs requires an MDS server and I think the only safe way to run that is active passive (I think active-active can lead to data corruption)
[23:09] <flaf> Question concerning cephfs in Infernalis: I think my cluster is OK and normally I have 3 running mds daemon with the defaut config (one is up and 2 are in standby). But with Infernalis, in the output of ???ceph -s???, the line ???mdsmap: xxx??? is absent (was present in Hammer). Is it normal? How can I check that mds are ok (with 1 up and 2 in standy) ? I have tried too ???ceph mds dump??? but the
[23:09] <flaf> output is different from Hammer. So, currently I'm not sure that all my mds are ok.
[23:13] * dack (~darrelle@gateway.ola.bc.ca) has joined #ceph
[23:13] * iPrime (~oftc-webi@host217-114-156-249.pppoe.mark-itt.net) Quit (Quit: Page closed)
[23:15] <dack> has anyone tried ceph cache tiering with SRM drives as the cold tier? if that works well it could really cut down on storage costs. not sure how it would handle rebalancing though.
[23:16] <dack> s/SRM/SMR
[23:21] <mfa298_> dack: We're using SMR drives (no cache tier yet) and I wouldn't recommend them with ceph, rebalancing has been slow and long at times
[23:21] <mfa298_> SSD journals with them help a bit but my initial tests of that still has them working slower than traditional drives
[23:22] <dack> mfa298_: how big of a cluster have you tried?
[23:22] <mfa298_> I'm hoping we might get our cluster to Health_OK in a few hours time, which will be the first time in about 3 weeks (having pulled some drives a few weeks back)
[23:22] <flaf> With Infernalis, I notice too that ???ceph health detail??? is not verbose at all (it was with Hammer). The output of this command is just 2 lines.
[23:23] * diq (~diq@2620:11c:f:2:c23f:d5ff:fe62:112c) Quit (Quit: Leaving)
[23:23] <mfa298_> dack: over 2000 SMR drives
[23:24] <dack> wow, that's pretty big. maybe the rebuilds wouldn't be so bad with smaller clusters?
[23:24] * musha68k (~zomg@h081217234161.dyn.cm.kabsi.at) has joined #ceph
[23:25] <mfa298_> for individual rebuilds it's going to be limited to a small number of individual drives.
[23:25] * diq (~diq@2620:11c:f:2:c23f:d5ff:fe62:112c) has joined #ceph
[23:26] <mfa298_> our recovery is down to 1 placement group and having just done the numbers it's recovering at 7.5 objects/s (objects being jpeg images)
[23:26] <mfa298_> last night it was much faster as it was recovering lots of placement groups in parallel
[23:26] <dack> wow, that is definitely quite slow...
[23:26] * georgem (~Adium@184.151.178.151) Quit (Read error: Connection reset by peer)
[23:27] * georgem (~Adium@206.108.127.16) has joined #ceph
[23:27] <m3porter> are you using replication or ec?
[23:28] <mfa298_> on a couple of smaller test clusters (one with smr, one with more traditional drives) doing a similar failure test the SMR test cluster was taking at least 3 times longer to recover (we stopped it before the SMR cluster had finally finished but it was close to done)
[23:28] * Venturi (Venturi@93-103-91-169.dynamic.t-2.net) Quit (Ping timeout: 480 seconds)
[23:28] <mfa298_> m3porter: standard replication (size=3, min_size=2)
[23:28] <m3porter> which SMR drives if i may ask?
[23:29] <dack> very interesting, thanks mfa298_
[23:29] <mfa298_> we've also had a couple of cascade failures which look to be related to the OSD maps being written to all the drives which then makes things slow to respond, so more map updates to be written ...
[23:29] * dgurtner (~dgurtner@185.10.235.250) Quit (Ping timeout: 480 seconds)
[23:29] <mfa298_> I'll post to the list about that at some point but I'm still trying to write it up.
[23:30] <mfa298_> m3porter: seagate 8TB archive drives
[23:31] * squizzi_ (~squizzi@nat-pool-rdu-u.redhat.com) has joined #ceph
[23:32] <dack> are you using XFS?
[23:33] <mfa298_> yes
[23:33] * squizzi (~squizzi@nat-pool-rdu-t.redhat.com) Quit (Ping timeout: 480 seconds)
[23:34] <mfa298_> I'd half like to test ext4 and possibly btrfs and see if either is better. I suspect ext4 won't make much difference, btrfs might be better but I'm not sure it would be wise to put that into production
[23:34] * CydeWeys (~tZ@bolobolo2.torservers.net) has joined #ceph
[23:34] <dack> i would think there would be significant performance improvements if the underlying FS was SMR-aware and using a host managed drive.
[23:34] <dack> as far as i know neither xfs or btrfs is there yet though.
[23:35] * georgem (~Adium@206.108.127.16) Quit (Quit: Leaving.)
[23:35] <dack> this could be interesting to test: https://github.com/Seagate/SMR_FS-EXT4
[23:35] <mfa298_> I don't think anything is there yet, These are also drive managed devices so I'm not sure any SMR aware filesystems will help a great deal
[23:37] <mfa298_> I've been watching that one, I think there's a bit to go before it's useful, tools look to be expected end of the month.
[23:37] <dack> yeah, i'm not sure if the seagate archive drives have any host-aware capabilities at all
[23:40] <mfa298_> I'm not sure if there are any host-aware or host managed drives yet - although they might be waiting on file system support.
[23:40] * Unai (~Adium@50-115-70-150.static-ip.telepacific.net) has joined #ceph
[23:41] <mfa298_> host managed in particular look like they won't be that useful until filesystems have caught up
[23:41] * masteroman (~ivan@93-142-202-73.adsl.net.t-com.hr) has joined #ceph
[23:43] <dack> i found it interesting that spectralogic is now advertising a product ("ArcticBlue") that is a ZFS array of seagate SMR disks.
[23:43] * masteroman (~ivan@93-142-202-73.adsl.net.t-com.hr) Quit ()
[23:44] <dack> i guess ZFS with a cache in front might be OK, as rebuilds would be pretty sequential
[23:46] <mfa298_> I've certainly seen suggestions that zfs can be good with them and I wonder if btrfs might also be better, at least in part to both of them being copy on write
[23:50] * jclm (~jclm@ip-64-134-184-248.public.wayport.net) has joined #ceph
[23:51] * jclm (~jclm@ip-64-134-184-248.public.wayport.net) Quit ()
[23:54] * herrsergio (~oftc-webi@200-77-224-239.kionetworks.com) has joined #ceph
[23:55] <herrsergio> hi everybody, I have a question regarding some log messages in a compute node and in a storage node. In the compute node there are some messages in syslog about "socket closed (con state OPEN)"
[23:55] <herrsergio> and in the storage node, in the ceph-osd.X.log there are some messages about "bad crc in data"
[23:55] <herrsergio> have you seen this kind of messages ?
[23:55] <herrsergio> the timestamps of the messages in syslog and in ceph-osd.X.log match

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.