#ceph IRC Log

Index

IRC Log for 2016-01-04

Timestamps are in GMT/BST.

[0:43] * olid1986 (~olid1982@aftr-185-17-206-52.dynamic.mnet-online.de) Quit (Ping timeout: 480 seconds)
[0:44] * wjw-freebsd (~wjw@smtp.digiware.nl) Quit (Ping timeout: 480 seconds)
[0:46] * wjw-freebsd (~wjw@smtp.digiware.nl) has joined #ceph
[1:00] * Nats (~natscogs@114.31.195.238) has joined #ceph
[1:00] <ben1> i've been playing with ceph, and i was surprised that gigabit wasn't actually that slow for client access, but at the same time i'm stlil only up to 750mb/sec read speed with faster network.
[1:01] <ben1> i am wondering if it's to do with 4mb block sizes and not doing parallel requests
[1:01] <ben1> is there any way to optimise such situations?
[1:01] <ben1> even 750mb/sec needed read-ahead boosted
[1:04] <ben1> ie have 16mb read ahead before kvm or such
[1:04] * fdmanana (~fdmanana@2001:8a0:6dfd:6d01:ed0c:dd2f:18a6:e24d) Quit (Ping timeout: 480 seconds)
[1:09] <doppelgrau> ben1: how many nodes with osds?
[1:09] <ben1> doppelgrau: just 3 for testing so far.
[1:09] <ben1> and only 10gbe between them
[1:10] <ben1> all have dual 10gbe cards directly connecting to each other
[1:10] <doppelgrau> platter or ssd?
[1:10] <ben1> ssd journal platter disk
[1:10] <ben1> but reads should fit in memory
[1:10] <doppelgrau> are you sure they are cached?
[1:10] <ben1> it doesn't look like ssds for data really is viable
[1:10] <ben1> oh?
[1:11] <ben1> will it not cache?
[1:11] <doppelgrau> uses the normal linux page cache
[1:11] <ben1> yeah
[1:11] <ben1> so should easily fit in memory
[1:12] <doppelgrau> ok, just wanted to be sure, that the cache is ???warm??? on all three nodes
[1:12] <ben1> yip
[1:13] <ben1> the disks are disbalanced..
[1:13] <ben1> 15k sas, and 10k sata, and 7.2k sata
[1:13] <ben1> i set the priority down on the 7.2k
[1:13] <ben1> and replicas is set to 2.
[1:14] <ben1> and i was testing using ddr infiniband off one
[1:14] <ben1> which can do more than 10 gigabit..
[1:14] <ben1> mtu is 9k
[1:14] <ben1> tcp_timestamps off, tcp_low_latency on
[1:14] <doppelgrau> what is your benachmark-tool?
[1:15] <ben1> the only thing i've found to make any noticable difference to performance is setting cstates.
[1:15] <ben1> bonnie on a kvm vm
[1:15] <ben1> on 4gb ram client, with 1gb vm
[1:15] <ben1> which should push it towards quick testing, and testing mostly peak performance :)
[1:15] <ben1> like if i do rados bench for a while my speeds drop..
[1:15] <ben1> with the whole journal thing
[1:16] <doppelgrau> hmm, so VM-Kernel =>API => qemu => librados ???
[1:16] <ben1> yeah
[1:16] <ben1> i think that sounds right
[1:16] <doppelgrau> how is the CPU-Usage at the host (especially qemu) while running the benachmark?
[1:16] <ben1> hmm i have to move it back onto infiniband again
[1:17] <ben1> atm i've been testing gigabit heh
[1:17] * doppelgrau has noticed that sometimes qemu can be quite cpu hungry
[1:17] <Freeaqingme> ben1: did you enable iothreads?
[1:17] <ben1> but it wasn't really high
[1:17] <Freeaqingme> in kvm?
[1:17] <ben1> Freeaqingme: i'm not sure
[1:17] <ben1> format=raw,file=rbd:rbd/ubuntu -device virtio-blk-pci,scsi=off,drive=drive0,vectors=6,\
[1:17] <ben1> config-wce=on,x-data-plane=on,disable-modern=off \
[1:17] <ben1> that's what i'm testing with atm
[1:17] * DV (~veillard@2001:41d0:a:f29f::1) Quit (Ping timeout: 480 seconds)
[1:17] <ben1> i was testing with scsi earlier
[1:18] <ben1> but scsi just gives higher write latency with block write
[1:18] <ben1> as it has 1024 nr_requests rathe than 128 nr_requests
[1:18] <Freeaqingme> try to enable iothreads. That should give an increase in IO performance
[1:18] <doppelgrau> ben1: try rados bench as a reference
[1:18] <ben1> i may be using iothreads
[1:18] <ben1> if=none,id=drive0,cache=none,aio=native,\
[1:19] <ben1> aio=native is iothreads isn't it?
[1:19] <ben1> trying to track down how this stuff works is insane
[1:19] <ben1> i'll go move it back :)
[1:19] <doppelgrau> ben1: that eliminates the KVM from the equation => if rados bench only results in 750 MB/s you have to take a look at the network/osds, else the source might be in the KVM-stack
[1:19] <ben1> oh hangon
[1:19] <ben1> rados bench was only testing writes not reads
[1:20] <Freeaqingme> ben1: I think what you're looking for is --iothreads=2 or something. At least I see <iothreads> in our kvm definitions
[1:20] * oms101 (~oms101@p20030057EA353800C6D987FFFE4339A1.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[1:20] <doppelgrau> ben1: you can tell rados bench to benchmark reads, it uses the data of a previous write benchmark that wasn???t deleted
[1:22] <ben1> what do people usually test with if testing at home anyway
[1:22] <ben1> i'm using mostly desktop hw :/
[1:22] <ben1> and it's still noisy
[1:22] <ben1> and i don't have spare pci-e slot for sas controller to use sas disks in one of the hosts
[1:23] * sankarshan (~sankarsha@106.216.172.128) has joined #ceph
[1:23] <doppelgrau> ben1: HW? I have some some small ???id 18??? boxes for playing at home,other stuff is done remotely
[1:24] <ben1> id 18?
[1:26] <doppelgrau> ben1: some variant of these: https://www.zotac.com/de/product/mini_pcs/id18-windows-81-bing (as barebone)
[1:28] * sankarshan (~sankarsha@106.216.172.128) Quit (Remote host closed the connection)
[1:28] <ben1> oh wow
[1:28] * oms101 (~oms101@p20030057EA11E700C6D987FFFE4339A1.dip0.t-ipconnect.de) has joined #ceph
[1:28] <ben1> i thought i was using low end gear
[1:28] * sankarshan (~sankarsha@106.216.172.128) has joined #ceph
[1:28] <doppelgrau> :)
[1:29] <ben1> i'm using one i3-3220 with 16gb ram
[1:29] <ben1> but it has serial console :))
[1:29] <ben1> but only one pci-e x16 slot
[1:29] <Freeaqingme> ben1: for what purposes are you testing? I've a colleague who used a bunch of vps's to test functionality
[1:29] <ben1> Freeaqingme: for vps usage.
[1:30] <ben1> well i want to see how well vps's work on it
[1:30] <ben1> so using real hw is better
[1:30] <Freeaqingme> just use a bunch of hw that you'd otherwise use in prod?
[1:30] <ben1> what i've found so far is that 7.2k rpm disks seem to suck when they have to catch up
[1:30] <Freeaqingme> must have lying some around, right?
[1:30] <ben1> Freeaqingme: it's all noisy!
[1:30] <ben1> atm i have stuff in my kitchen
[1:30] <Freeaqingme> ahlol
[1:31] <ben1> but even then it's pretty noisy
[1:31] <ben1> and i'm kind of as hot as i want to make it in there
[1:31] <Freeaqingme> no basement or something?
[1:31] <ben1> yeah no basement or anything
[1:31] <doppelgrau> I use then only for ???toying??? and testing concepts (do not need much room and are quite), in the moment I want serious performance results I usually switch to dedicated HW somewhre else, but at that stage I usually do not need serial ports or access to the console
[1:31] <ben1> ahh
[1:32] <Freeaqingme> I got the luxury that I can submit a ticket that I need a rack with 10 24" SSD machines, and some blades, and it's done in a few days. That's a luxury not everybody has though
[1:32] <ben1> i'm liking these hp 6300 sffs
[1:32] <ben1> Freeaqingme: haha yes i don't hav that
[1:33] <ben1> i'm still a bit weirded out by how much slower the 7.2k sata disks are compared to the 10k sata
[1:33] <ben1> the 10k sata is closer to 15k sas then to 7.2k sata
[1:33] <ben1> sure the sas drives are only cheetah 15k.6 which are old
[1:33] <ben1> but i happened across these velociraptors and they seek really fast
[1:34] <ben1> and handle parallel tasks about twice as fast as the 7.2k disks
[1:34] <Freeaqingme> ben1: SSD's not an option? They did get quite a lot cheaper the past few years
[1:34] <ben1> maybe it's just a firmware thing
[1:34] <ben1> Freeaqingme: i have some ssd's..
[1:34] <Freeaqingme> I mean for data as well
[1:35] <ben1> but i'd need to have a journal per ssd?
[1:35] <ben1> so ssd journal, ssd data?
[1:35] * elder_ (~elder@c-24-61-14-142.hsd1.ma.comcast.net) Quit (Quit: Leaving)
[1:35] <Freeaqingme> well, if you want performance
[1:35] <ben1> it shouldn't help my read speeds
[1:35] <ben1> as long as i can fit in cache
[1:35] <Freeaqingme> true
[1:36] <ben1> the write speeds thing i'm curious about though
[1:36] * doppelgrau had some disks where the firmware got mad sometimes and for some days the performance doped neary by half (utilization doubled), after some times it retourned to normal only to return while later
[1:36] <ben1> like i'd like to be able to get ssd write speed always
[1:36] <ben1> but instead i get ssd write speed, then it writes to both at once, then it keeps writing to the second one
[1:36] <doppelgrau> was ugly to find the reason why sometimes you had one osd with tons of slow requests
[1:36] <ben1> and if i can shift that writing to the disk to after the test finishes it'd be nice :)
[1:36] <ben1> rather than slowing down to the disk speed
[1:37] <ben1> heh some of my disks are slightly different speeds
[1:37] <ben1> even when they're the same specs
[1:37] <ben1> it'd be nice if that kind of thing could just be dynamically dealt with
[1:38] <Freeaqingme> ben1: smart data doesn't help you either?
[1:38] <ben1> it's actually the random read performance i want to boost
[1:38] <ben1> Freeaqingme: np[e
[1:38] <ben1> nope
[1:38] <ben1> seagate used to have huge discrepencies in some drives
[1:39] <ben1> becuase they're use different numbers of platters in drives with the same model number
[1:39] <ben1> but these are closer than that
[1:39] <Freeaqingme> heh yeah
[1:39] <Freeaqingme> long live seagate
[1:39] <ben1> hey
[1:39] <ben1> the 15k seagate sas disks work really well
[1:40] <ben1> actually they're HP DF0450B8054
[1:40] <Freeaqingme> yeah, I'm not dissing seagate. It's just the pranks they pull with the # of platters etc.
[1:40] <ben1> oh right
[1:40] <ben1> yeah i hate the hard-disk market
[1:41] <ben1> but the ssd market isn't any better
[1:41] <ben1> wdc and seagate are both bad
[1:41] <Freeaqingme> you're sure they're all seagates? I've had a bunch of dell drives in the past that were all rebranded to dell, but within the same series I had multiple manufacturers
[1:41] <ben1> then ssd's there's all kinds of random stuff
[1:41] <ben1> no i'm not sure they're all seagate
[1:41] <ben1> is there an easy way to check without pulling them out?
[1:42] <Freeaqingme> seagate had really good ssd's, until hynix got sold iirc. And they cancelled all their ssd's in 2 weeks.
[1:42] <Freeaqingme> these days intel and toshiba are pretty okay imho
[1:42] <ben1> i was thinking hdd
[1:42] <ben1> intel ssd's with power loss protection seem expensive
[1:42] <Freeaqingme> I /think/ there should be some manufacturer tag in the smart stuff?
[1:42] <ben1> but i was reading that samsung ssd's may go back in time on power fail?
[1:43] <ben1> not in my smart
[1:43] <ben1> it hardly shows anything in smart compared to sata disks
[1:43] <Freeaqingme> dunno about samsung ssd's. Tried them a while back, but their garbage collector only worked on the foreground. And with continuous load, they never got to do any garbage collection
[1:43] <ben1> wel;l it tells me the disk has 16.5TB written
[1:43] <ben1> so it's not exactly new :)
[1:43] <Freeaqingme> oh, that's a pity
[1:44] <Freeaqingme> I have never really used sas hdd's. Only Sata ones
[1:44] <ben1> one of them is 18.5TB
[1:44] <ben1> they have really high reads done too
[1:44] <ben1> like 130tb etc
[1:44] <ben1> which is strange to my mind, cos all thje workloads i see are higher write then read
[1:44] <ben1> s/then/than/
[1:45] <ben1> because system ram read caching etc these days
[1:45] <Freeaqingme> maybe your load was different once you started using them?
[1:45] <Freeaqingme> anyways, you could do some local benchmarking first, see if you can rule out hdd performance?
[1:45] <Freeaqingme> or did you already do that?
[1:46] <ben1> i tested all the disks
[1:46] <ben1> 120 through 190mb/sec
[1:46] <ben1> for sequential
[1:46] <ben1> and uhh 56 to 260 iops/sec
[1:47] <ben1> but that was using ioping
[1:47] <ben1> and ioping seems to be pretty bad for testing that
[1:48] <ben1> min/avg/max/mdev = 5.39 ms / 15.2 ms / 26.3 ms / 4.26 ms
[1:48] <ben1> min/avg/max/mdev = 1.41 ms / 5.75 ms / 10.4 ms / 1.72 ms
[1:48] <ben1> i'm not sure what's normal there
[1:48] <ben1> but 15msec average access time seems insane
[1:48] <ben1> and 5.75 just looks ok?
[1:48] <Freeaqingme> I dunno
[1:48] <ben1> ssd was WAY better though
[1:49] <ben1> min/avg/max/mdev = 17 us / 34 us / 283 us / 23 us
[1:50] <ben1> could be power state stuff there though, with ssd responding quick enough that cpu doesn't go to sleep
[1:50] <Freeaqingme> then you disable all the power stuff?
[1:50] <Freeaqingme> at least for testing
[1:51] <Freeaqingme> I never tested the difference, but I do know that for some reason all our procedures state that it should be disabled
[1:54] <ben1> yeh i did
[1:54] <ben1> then i reenabled it to reduce power
[1:54] <Freeaqingme> oh yeah, the heat
[1:54] <ben1> there's this cool program i found online that you just run it to set desired latency
[1:55] <ben1> https://github.com/gtcasl/hpc-benchmarks/blob/master/setcpulatency.c
[1:56] <ben1> that's the same diff when goolging for it
[1:56] <ben1> so i find a setting of 1 or 2 gives good low latency
[1:56] <ben1> and increases power usage by about 20 watts or so
[1:57] <ben1> well on e5620, which is the only server that i have with power monitoring.
[1:58] <ben1> it mostly helped for latency, throughput was only about 20mb/sec faster.
[1:59] * linjan_ (~linjan@176.195.50.55) Quit (Ping timeout: 480 seconds)
[1:59] * vbellur (~vijay@2601:647:4f00:4960:5e51:4fff:fee8:6a5c) Quit (Quit: Leaving.)
[2:00] * vbellur (~vijay@2601:647:4f00:4960:5e51:4fff:fee8:6a5c) has joined #ceph
[2:01] * LeaChim (~LeaChim@host86-132-236-140.range86-132.btcentralplus.com) Quit (Ping timeout: 480 seconds)
[2:31] * yanzheng (~zhyan@182.139.23.32) has joined #ceph
[3:11] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[3:21] * derjohn_mobi (~aj@x590ceea9.dyn.telefonica.de) has joined #ceph
[3:28] * naoto (~naotok@27.131.11.254) has joined #ceph
[3:29] * derjohn_mob (~aj@x4db11e4b.dyn.telefonica.de) Quit (Ping timeout: 480 seconds)
[3:29] * xarses_ (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) has joined #ceph
[3:31] * rendar (~I@host65-178-dynamic.49-79-r.retail.telecomitalia.it) Quit (Quit: std::lower_bound + std::less_equal *works* with a vector without duplicates!)
[3:33] * haomaiwa_ (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) has joined #ceph
[3:35] * dlan (~dennis@116.228.88.131) Quit (Quit: leaving)
[3:36] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Ping timeout: 480 seconds)
[3:49] * doppelgrau_ (~doppelgra@p5DC07B91.dip0.t-ipconnect.de) has joined #ceph
[3:55] * doppelgrau (~doppelgra@p5DC06FD9.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[3:55] * doppelgrau_ is now known as doppelgrau
[4:01] * haomaiwa_ (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) Quit (Remote host closed the connection)
[4:01] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) has joined #ceph
[4:05] * Inuyasha (~Kakeru@6YRAABVKI.tor-irc.dnsbl.oftc.net) has joined #ceph
[4:05] * zhaochao (~zhaochao@111.161.77.227) has joined #ceph
[4:05] * DV (~veillard@2001:41d0:a:f29f::1) has joined #ceph
[4:07] * dlan (~dennis@116.228.88.131) has joined #ceph
[4:19] * shylesh__ (~shylesh@121.244.87.124) has joined #ceph
[4:34] * Inuyasha (~Kakeru@6YRAABVKI.tor-irc.dnsbl.oftc.net) Quit ()
[4:35] * lcurtis (~lcurtis@ool-44c08556.dyn.optonline.net) has joined #ceph
[4:40] * kefu (~kefu@114.92.107.250) has joined #ceph
[5:01] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) Quit (Remote host closed the connection)
[5:01] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) has joined #ceph
[5:06] * amote (~amote@121.244.87.116) has joined #ceph
[5:11] * yanzheng (~zhyan@182.139.23.32) Quit (Quit: This computer has gone to sleep)
[5:17] * MACscr (~Adium@2601:247:4101:a0be:14ca:756a:f7ec:2ca0) Quit (Quit: Leaving.)
[5:35] * jith (~chatzilla@14.139.180.40) has joined #ceph
[5:36] <jith> hi all, Is there any GUI/Dashboard for ceph to access through browser?
[5:39] <[arx]> calamari is one of the more known ones, but there are others.
[5:40] <motk> anyone know if infernalis radosgw still honours old style bucket urls?
[5:47] * sankarshan (~sankarsha@106.216.172.128) Quit (Quit: Leaving...)
[5:49] * MACscr (~Adium@2601:247:4101:a0be:d0c:67bc:e3ba:32f0) has joined #ceph
[5:52] * Vacuum_ (~Vacuum@88.130.202.226) has joined #ceph
[5:56] * kanagaraj (~kanagaraj@121.244.87.117) has joined #ceph
[5:56] <doppelgrau> jith: for status I like cephdash (lightweight)
[5:57] <jith> doppelgrau: thanks
[5:57] <jith> [arx]: thanks
[5:58] <jith> [arx] , doppelgrau : what abt kraken.. http://ceph.com/planet/kraken-the-first-free-ceph-dashboard-in-town/??
[5:59] <doppelgrau> never tried
[5:59] * Vacuum__ (~Vacuum@88.130.210.132) Quit (Ping timeout: 480 seconds)
[6:00] * jamespage (~jamespage@2a00:1098:0:80:1000:42:0:1) Quit (Quit: Coyote finally caught me)
[6:00] * jamespage (~jamespage@2a00:1098:0:80:1000:42:0:1) has joined #ceph
[6:01] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) Quit (Remote host closed the connection)
[6:01] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) has joined #ceph
[6:15] * wjw-freebsd (~wjw@smtp.digiware.nl) Quit (Ping timeout: 480 seconds)
[6:18] * branto1 (~branto@178-253-138-162.3pp.slovanet.sk) has joined #ceph
[6:19] * efirs (~firs@c-73-231-190-117.hsd1.ca.comcast.net) has joined #ceph
[6:27] * rdas (~rdas@121.244.87.116) has joined #ceph
[6:29] * krypto (~krypto@125.16.137.146) has joined #ceph
[6:33] * swami1 (~swami@49.32.0.146) has joined #ceph
[6:41] * overclk (~vshankar@121.244.87.117) has joined #ceph
[6:58] * lcurtis (~lcurtis@ool-44c08556.dyn.optonline.net) Quit (Ping timeout: 480 seconds)
[7:01] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) Quit (Remote host closed the connection)
[7:01] * krypto (~krypto@125.16.137.146) Quit (Ping timeout: 480 seconds)
[7:01] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) has joined #ceph
[7:02] * krypto (~krypto@G68-121-13-160.sbcis.sbc.com) has joined #ceph
[7:04] * m0zes (~mozes@beocat.cis.ksu.edu) Quit (Quit: WeeChat 1.3)
[7:11] * trociny (~mgolub@93.183.239.2) Quit (Quit: ??????????????)
[7:11] * trociny (~mgolub@93.183.239.2) has joined #ceph
[7:11] * trociny (~mgolub@93.183.239.2) Quit ()
[7:12] * trociny (~mgolub@93.183.239.2) has joined #ceph
[7:15] * xiangxinyong (~xiangxiny@182.138.104.21) has joined #ceph
[7:33] * yanzheng (~zhyan@182.139.23.32) has joined #ceph
[7:36] * pepzi (~Sophie@178.162.216.42) has joined #ceph
[7:43] * i_m (~ivan.miro@88.206.113.199) has joined #ceph
[7:43] * karnan (~karnan@121.244.87.117) has joined #ceph
[7:54] * wer (~wer@216.197.66.226) has joined #ceph
[7:59] * Be-El (~quassel@fb08-bcf-pc01.computational.bio.uni-giessen.de) has joined #ceph
[8:00] <Be-El> hi
[8:01] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) Quit (Remote host closed the connection)
[8:01] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) has joined #ceph
[8:06] * pepzi (~Sophie@84ZAAA1VR.tor-irc.dnsbl.oftc.net) Quit ()
[8:12] * rotbeard (~redbeard@185.32.80.238) has joined #ceph
[8:31] * rakeshgm (~rakesh@121.244.87.117) has joined #ceph
[8:32] * rwheeler (~rwheeler@bzq-82-81-161-51.red.bezeqint.net) has joined #ceph
[8:35] * T1w (~jens@node3.survey-it.dk) has joined #ceph
[8:39] * owasserm (~owasserm@2001:984:d3f7:1:5ec5:d4ff:fee0:f6dc) has joined #ceph
[8:44] * ade (~abradshaw@dslb-092-078-128-021.092.078.pools.vodafone-ip.de) has joined #ceph
[8:47] * Kioob`Taff (~plug-oliv@2a01:e35:2e8a:1e0::42:10) has joined #ceph
[8:49] * enax (~enax@hq.ezit.hu) has joined #ceph
[8:55] * doppelgrau (~doppelgra@p5DC07B91.dip0.t-ipconnect.de) Quit (Quit: doppelgrau)
[8:55] * doppelgrau (~doppelgra@p5DC07B91.dip0.t-ipconnect.de) has joined #ceph
[8:58] * Kioob`Taff (~plug-oliv@2a01:e35:2e8a:1e0::42:10) Quit (Quit: Leaving.)
[8:58] * Kioob`Taff (~plug-oliv@2a01:e35:2e8a:1e0::42:10) has joined #ceph
[9:01] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) Quit (Remote host closed the connection)
[9:01] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) has joined #ceph
[9:01] * pabluk_ is now known as pabluk
[9:04] * derjohn_mobi (~aj@x590ceea9.dyn.telefonica.de) Quit (Ping timeout: 480 seconds)
[9:05] * analbeard (~shw@support.memset.com) has joined #ceph
[9:05] * dugravot6 (~dugravot6@dn-infra-04.lionnois.univ-lorraine.fr) has joined #ceph
[9:07] * daviddcc (~dcasier@84.197.151.77.rev.sfr.net) Quit (Ping timeout: 480 seconds)
[9:12] * efirs (~firs@c-73-231-190-117.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[9:13] * kefu (~kefu@114.92.107.250) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[9:16] * kefu (~kefu@183.193.128.154) has joined #ceph
[9:19] * jordanP (~jordan@pas38-2-82-67-72-49.fbx.proxad.net) has joined #ceph
[9:26] * dgurtner (~dgurtner@178.197.231.6) has joined #ceph
[9:30] * pepzi (~Tarazed@tor-relay.zwiebeltoralf.de) has joined #ceph
[9:32] * wjw-freebsd (~wjw@smtp.digiware.nl) has joined #ceph
[9:37] * thomnico (~thomnico@2a01:e35:8b41:120:48c4:8169:429f:b1ce) has joined #ceph
[9:39] * DanFoster (~Daniel@2a00:1ee0:3:1337:613d:7a75:fb8a:b1b) has joined #ceph
[9:40] * allaok (~allaok@machine107.orange-labs.com) has joined #ceph
[9:43] * aliriza (~oftc-webi@91.75.74.12) has joined #ceph
[9:43] <aliriza> Hi, I am trying to use rbd-nbd command as documented here http://docs.ceph.com/docs/master/man/8/rbd-nbd/
[9:44] <aliriza> but resultis "rbd-nbd: command not found"
[9:44] <aliriza> is there anyone who managed to use nbd functionality?
[9:45] <Be-El> aliriza: check that your ceph version corresponds to the documentation (hint: the "master" branch refers to git head...) and install the correct package for your linux distribution
[9:45] <aliriza> Thank you
[9:46] <aliriza> I have installed the infernalis version
[9:46] <aliriza> result of ceph --version is
[9:46] <aliriza> ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
[9:46] * nigwil (~Oz@li1101-124.members.linode.com) Quit (Read error: Connection reset by peer)
[9:46] <aliriza> I used ceph-deploy for the installation on the whole cluster
[9:47] <aliriza> and clients
[9:48] <aliriza> am I still missing something?
[9:48] * kefu (~kefu@183.193.128.154) Quit (Max SendQ exceeded)
[9:49] * dgurtner (~dgurtner@178.197.231.6) Quit (Quit: leaving)
[9:50] <Be-El> infernalis does not contain rbd-nbd according to its documentation (http://docs.ceph.com/docs/infernalis/rbd/rbd/)
[9:50] <aliriza> thank you Be-El
[9:50] <aliriza> after your comment, I tried the link and it doesn't contain
[9:51] * dgurtner (~dgurtner@178.197.231.6) has joined #ceph
[9:51] <aliriza> so, only option is to compile for this command
[9:51] * nigwil (~Oz@li1101-124.members.linode.com) has joined #ceph
[9:52] * nigwil (~Oz@li1101-124.members.linode.com) Quit (Read error: Connection reset by peer)
[9:52] <Be-El> if you really need it. the standard rbd commands also allows you to access rbd images as block devices
[9:52] * swami1 (~swami@49.32.0.146) Quit (Quit: Leaving.)
[9:52] <Be-El> do you have an particular reason to use the nbd implementation?
[9:53] * swami1 (~swami@49.44.57.238) has joined #ceph
[9:56] * nigwil (~Oz@li1101-124.members.linode.com) has joined #ceph
[9:57] * vikhyat (~vumrao@121.244.87.116) has joined #ceph
[9:59] * kefu (~kefu@211.22.145.245) has joined #ceph
[10:00] * nigwil (~Oz@li1101-124.members.linode.com) Quit (Read error: Connection reset by peer)
[10:00] * pepzi (~Tarazed@76GAAAZHO.tor-irc.dnsbl.oftc.net) Quit ()
[10:01] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) Quit (Remote host closed the connection)
[10:01] * sankarshan (~sankarsha@106.216.172.128) has joined #ceph
[10:01] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) has joined #ceph
[10:01] * nigwil (~Oz@li1101-124.members.linode.com) has joined #ceph
[10:02] * fsimonce (~simon@host229-72-dynamic.54-79-r.retail.telecomitalia.it) has joined #ceph
[10:03] * nigwil (~Oz@li1101-124.members.linode.com) Quit ()
[10:05] * xanax` (~isaxi@108.61.123.83) has joined #ceph
[10:06] * rakeshgm (~rakesh@121.244.87.117) Quit (Ping timeout: 480 seconds)
[10:11] * kefu_ (~kefu@183.193.128.154) has joined #ceph
[10:12] * TMM (~hp@sams-office-nat.tomtomgroup.com) has joined #ceph
[10:14] * rakeshgm (~rakesh@121.244.87.124) has joined #ceph
[10:14] * nardial (~ls@dslb-088-072-094-077.088.072.pools.vodafone-ip.de) has joined #ceph
[10:15] * kefu (~kefu@211.22.145.245) Quit (Ping timeout: 480 seconds)
[10:16] * steveeJ (~junky@HSI-KBW-149-172-252-139.hsi13.kabel-badenwuerttemberg.de) has joined #ceph
[10:22] * Kvisle (~tv@tv.users.bitbit.net) Quit (Remote host closed the connection)
[10:26] <aliriza> Sorry Be-El, I was dragged to an ad-hoc meeting
[10:26] * ngoswami (~ngoswami@121.244.87.116) has joined #ceph
[10:26] <aliriza> I am using nbd to change the partition size within the VM without needing to boot the VM with a live CD or so
[10:26] * circ-user-rwzoV (~circuser-@sessfw99-sesbfw99-91.ericsson.net) has joined #ceph
[10:26] <aliriza> this is the only use I am planning to utilize nbd
[10:27] <aliriza> Before ceph, I was able to use qemu-nbd to map the image files, but now I am out of options according to my knowledge
[10:27] * fdmanana (~fdmanana@2001:8a0:6dfd:6d01:ed0c:dd2f:18a6:e24d) has joined #ceph
[10:27] <circ-user-rwzoV> Hi guys, I have an issue with ceph I hope you can solve
[10:28] <aliriza> as you mentioned, I can give it a try to mount as kernel rbd devices and make the changes
[10:29] <aliriza> It unfortunately means some additional development time to migrate to ceph fully.
[10:29] <circ-user-rwzoV> I have an app that writes to a ceph pool successfully with the "aio_append" command of the rados api. I have now changed the pool to erasure-coded pool.
[10:30] <circ-user-rwzoV> Now the append command doesn't work anymore. I can only write once, and can not append data
[10:31] <circ-user-rwzoV> I would really appreciate any help with that
[10:32] <circ-user-rwzoV> anyone ?
[10:34] <Be-El> aliriza: do you want to resize a partition of an active rbd device?
[10:34] * xanax` (~isaxi@108.61.123.83) Quit ()
[10:35] <IcePic> circ-user-rwzoV: are you sure nothing else changed at the same time?
[10:35] <circ-user-rwzoV> Yes
[10:36] <circ-user-rwzoV> I have a normal pool which works
[10:36] <Be-El> circ-user-rwzoV: there are some limitations to ec pools. i'm not sure whether appending works at all for chunks blocks than the stripe size
[10:36] <circ-user-rwzoV> And I have an erasure-coded pool which I can write to an object only once
[10:36] <Be-El> circ-user-rwzoV: that's also the reason EC pools cannot be usd for rbd or cephfs directly
[10:36] <aliriza> Be-El: no. Changing the partition table will occur while the creation of the new VM.
[10:36] <Be-El> circ-user-rwzoV: you can try to put a cache layer in front of the ec pool
[10:37] <circ-user-rwzoV> Would that influence benchamrks ?
[10:37] <circ-user-rwzoV> *Benchmarks
[10:38] <circ-user-rwzoV> How do I do that ? And is the meaning of that is that after the cache layer resets, I would not be able to write to that object again ?
[10:38] <Be-El> aliriza: in that case the standard rbd tool should work more or less exactly as rbd-nbd
[10:38] * kefu_ is now known as kefu|afk
[10:39] <Be-El> circ-user-rwzoV: the idea of the cache layer in that scenario is to allow you to modify the object afterwards
[10:39] * Thononain (~Corti^car@37.48.81.27) has joined #ceph
[10:40] <Be-El> circ-user-rwzoV: but I thought that EC pools allow to append to an object, so you might better want to wait for someone with more experience with the mentioned rados call
[10:40] <aliriza> Be-El: I am literraly giving it a try now
[10:40] <aliriza> and you certainly are right
[10:40] <aliriza> thank you for pointing to the right direction.
[10:40] <Be-El> aliriza: there may be some easier way
[10:41] <aliriza> Be-El: can you please elaborate if you have something in your mind?
[10:41] <Be-El> aliriza: do you know libguestfish?
[10:41] <aliriza> Be-El: I do, but I am not very happy with the tools
[10:41] * kefu|afk is now known as kefu_
[10:41] <Be-El> aliriza: ah, ok
[10:42] * kefu_ is now known as kefu
[10:42] <aliriza> Be-El: thank you for brilliant ideas.
[10:42] <Be-El> aliriza: you're welcome
[10:42] * yankcrime (~yankcrime@185.43.216.241) has joined #ceph
[10:43] <circ-user-rwzoV> Be-El: Do you know of some1 that might be helpful with the rados api ?
[10:43] <Be-El> circ-user-rwzoV: the devel channel might be better suited for that question
[10:44] <circ-user-rwzoV> I'll give it a try, thanks !
[10:46] * aliriza (~oftc-webi@91.75.74.12) Quit (Quit: Page closed)
[10:48] * yanzheng (~zhyan@182.139.23.32) Quit (Quit: This computer has gone to sleep)
[11:01] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) Quit (Remote host closed the connection)
[11:01] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) has joined #ceph
[11:01] * nigwil (~Oz@li1416-21.members.linode.com) has joined #ceph
[11:02] * lkoranda (~lkoranda@nat-pool-brq-t.redhat.com) Quit (Ping timeout: 480 seconds)
[11:03] * LeaChim (~LeaChim@host86-132-236-140.range86-132.btcentralplus.com) has joined #ceph
[11:05] * derjohn_mobi (~aj@2001:6f8:1337:0:612a:ee5f:ae4:64aa) has joined #ceph
[11:06] * yanzheng (~zhyan@182.139.23.32) has joined #ceph
[11:09] * Thononain (~Corti^car@4MJAAAZQM.tor-irc.dnsbl.oftc.net) Quit ()
[11:09] * kefu (~kefu@183.193.128.154) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[11:12] * olid1986 (~olid1982@aftr-185-17-204-109.dynamic.mnet-online.de) has joined #ceph
[11:17] * Mika_c (~Mika@122.146.93.152) has joined #ceph
[11:19] * InIMoeK (~InIMoeK@95.170.93.16) has joined #ceph
[11:24] * circ-user-rwzoV (~circuser-@sessfw99-sesbfw99-91.ericsson.net) Quit (Ping timeout: 480 seconds)
[11:26] * yanzheng (~zhyan@182.139.23.32) Quit (Quit: This computer has gone to sleep)
[11:46] * ngoswami (~ngoswami@121.244.87.116) Quit (Quit: Leaving)
[11:48] * ngoswami (~ngoswami@121.244.87.116) has joined #ceph
[11:49] * circ-user-rwzoV (~circuser-@sessfw99-sesbfw99-91.ericsson.net) has joined #ceph
[11:52] * kefu (~kefu@183.193.128.154) has joined #ceph
[11:52] * rogierm (~rogierm@2001:985:1c56:1:5cc8:d00c:5944:8ddc) has joined #ceph
[11:55] * rendar (~I@87.19.183.44) has joined #ceph
[12:00] * kefu (~kefu@183.193.128.154) Quit (Max SendQ exceeded)
[12:01] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) Quit (Remote host closed the connection)
[12:01] * kefu (~kefu@211.22.145.245) has joined #ceph
[12:01] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) has joined #ceph
[12:04] * kefu is now known as kefu|afk
[12:08] * Mika_c (~Mika@122.146.93.152) Quit (Read error: Connection reset by peer)
[12:13] <Heebie> Has anyone used ceph with ProxMox? I'm trying to get a ProxMox node connected, and I created a user with mon "allow r" and osd "allow class-read object_prefix rbd_children, allow rwx pool=ProxMoxTest" (and the pool exists) I'm getting "Operation not permitted" and "Error connecting to cluster: PermissionError"
[12:14] * aliriza (~oftc-webi@91.75.74.12) has joined #ceph
[12:18] * rogierm (~rogierm@2001:985:1c56:1:5cc8:d00c:5944:8ddc) Quit (Remote host closed the connection)
[12:18] * rwheeler (~rwheeler@bzq-82-81-161-51.red.bezeqint.net) Quit (Quit: Leaving)
[12:21] * Be-El (~quassel@fb08-bcf-pc01.computational.bio.uni-giessen.de) Quit (Ping timeout: 480 seconds)
[12:23] * rogierm (~rogierm@2001:985:1c56:1:c1b:67b8:423:e4a6) has joined #ceph
[12:24] * sankarshan (~sankarsha@106.216.172.128) Quit (Quit: Leaving...)
[12:25] <Jeeves_> Heebie: I do that
[12:26] <Jeeves_> Heebie: Do you have /etc/pve/priv/ceph/ProxMoxTest.keyring ?
[12:27] * daviddcc (~dcasier@LAubervilliers-656-1-16-160.w217-128.abo.wanadoo.fr) has joined #ceph
[12:36] * kefu|afk (~kefu@211.22.145.245) Quit (Read error: Connection reset by peer)
[12:38] * kefu (~kefu@183.193.128.154) has joined #ceph
[12:42] * Sophie1 (~jwandborg@185.101.107.136) has joined #ceph
[12:42] * naoto (~naotok@27.131.11.254) Quit (Quit: Leaving...)
[12:45] * rogierm (~rogierm@2001:985:1c56:1:c1b:67b8:423:e4a6) Quit (Remote host closed the connection)
[12:46] * rogierm_ (~rogierm@a82-94-41-183.adsl.xs4all.nl) has joined #ceph
[12:46] * kefu (~kefu@183.193.128.154) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[12:55] * rogierm_ (~rogierm@a82-94-41-183.adsl.xs4all.nl) Quit (Remote host closed the connection)
[12:56] * yanzheng (~zhyan@182.139.23.32) has joined #ceph
[12:57] * Be-El (~quassel@fb08-bcf-pc01.computational.bio.uni-giessen.de) has joined #ceph
[12:58] * rogierm (~rogierm@2001:985:1c56:1:b957:1108:8188:cddb) has joined #ceph
[13:00] * shylesh__ (~shylesh@121.244.87.124) Quit (Remote host closed the connection)
[13:01] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) Quit (Remote host closed the connection)
[13:01] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) has joined #ceph
[13:01] * nigwil (~Oz@li1416-21.members.linode.com) Quit (Remote host closed the connection)
[13:01] * krypto (~krypto@G68-121-13-160.sbcis.sbc.com) Quit (Quit: Leaving)
[13:06] * rakeshgm (~rakesh@121.244.87.124) Quit (Ping timeout: 480 seconds)
[13:08] * zhaochao (~zhaochao@111.161.77.227) Quit (Quit: ChatZilla 0.9.92 [Iceweasel 43.0.2/20151228074541])
[13:12] * Sophie1 (~jwandborg@4MJAAAZTA.tor-irc.dnsbl.oftc.net) Quit ()
[13:12] * Thononain (~Snowman@176.61.147.146) has joined #ceph
[13:15] * rakeshgm (~rakesh@121.244.87.117) has joined #ceph
[13:17] * rogierm (~rogierm@2001:985:1c56:1:b957:1108:8188:cddb) Quit (Remote host closed the connection)
[13:19] * rogierm (~rogierm@2001:985:1c56:1:b08d:71a1:61e2:bed6) has joined #ceph
[13:23] * bene_in_mtg (~bene@2601:18c:8300:f3ae:ea2a:eaff:fe08:3c7a) has joined #ceph
[13:24] * bene_in_mtg (~bene@2601:18c:8300:f3ae:ea2a:eaff:fe08:3c7a) Quit ()
[13:42] * Thononain (~Snowman@4MJAAAZT3.tor-irc.dnsbl.oftc.net) Quit ()
[13:47] * rwheeler (~rwheeler@5.29.243.114) has joined #ceph
[13:49] * kefu (~kefu@114.92.107.250) has joined #ceph
[13:50] * wjw-freebsd (~wjw@smtp.digiware.nl) Quit (Ping timeout: 480 seconds)
[13:51] * rogierm (~rogierm@2001:985:1c56:1:b08d:71a1:61e2:bed6) Quit (Remote host closed the connection)
[13:52] * overclk (~vshankar@121.244.87.117) Quit (Quit: Zzzz...)
[13:54] * T1w (~jens@node3.survey-it.dk) Quit (Ping timeout: 480 seconds)
[13:55] * alfredodeza (~alfredode@198.206.133.89) has joined #ceph
[14:01] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) Quit (Remote host closed the connection)
[14:01] * aliriza (~oftc-webi@91.75.74.12) has left #ceph
[14:01] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) has joined #ceph
[14:03] * todin (tuxadero@kudu.in-berlin.de) has joined #ceph
[14:06] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) Quit (Remote host closed the connection)
[14:08] * wjw-freebsd (~wjw@vpn.ecoracks.nl) has joined #ceph
[14:08] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) has joined #ceph
[14:16] * LDA (~DM@host217-114-156-249.pppoe.mark-itt.net) has joined #ceph
[14:18] * amote (~amote@121.244.87.116) Quit (Ping timeout: 480 seconds)
[14:24] * yanzheng (~zhyan@182.139.23.32) Quit (Quit: This computer has gone to sleep)
[14:28] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) Quit (Remote host closed the connection)
[14:29] * rdas (~rdas@121.244.87.116) Quit (Quit: Leaving)
[14:29] * circ-user-rwzoV (~circuser-@sessfw99-sesbfw99-91.ericsson.net) Quit (Ping timeout: 480 seconds)
[14:30] * vikhyat (~vumrao@121.244.87.116) Quit (Quit: Leaving)
[14:33] * rakeshgm (~rakesh@121.244.87.117) Quit (Quit: Leaving)
[14:36] * yanzheng (~zhyan@182.139.23.32) has joined #ceph
[14:36] * kanagaraj (~kanagaraj@121.244.87.117) Quit (Quit: Leaving)
[14:37] * treenerd (~treenerd@85.193.140.98) has joined #ceph
[14:37] * karnan (~karnan@121.244.87.117) Quit (Remote host closed the connection)
[14:38] * branto1 (~branto@178-253-138-162.3pp.slovanet.sk) Quit (Quit: Leaving.)
[14:39] * dugravot6 (~dugravot6@dn-infra-04.lionnois.univ-lorraine.fr) Quit (Quit: Leaving.)
[14:42] * jrankin (~jrankin@d53-64-170-236.nap.wideopenwest.com) has joined #ceph
[14:44] * dugravot6 (~dugravot6@dn-infra-04.lionnois.univ-lorraine.fr) has joined #ceph
[14:47] * circ-user-rwzoV (~circuser-@sessfw99-sesbfw99-91.ericsson.net) has joined #ceph
[14:49] * RMar04 (~RMar04@support.memset.com) has joined #ceph
[14:50] * jcsp_ (~jspray@82-71-16-249.dsl.in-addr.zen.co.uk) Quit (Quit: Ex-Chat)
[14:51] * allaok (~allaok@machine107.orange-labs.com) has left #ceph
[14:53] * Rachana (~Rachana@2601:87:3:3601::7554) has joined #ceph
[14:53] * Grimhound (~FierceFor@185.100.85.132) has joined #ceph
[14:54] <circ-user-rwzoV> Hey guys. I've made an erasure-coded pool with a replicated cache tier. I have two questions though :
[14:54] <circ-user-rwzoV> 1. If a file's size let's say, is 50% of the storage, and the cache tier is configured to write to the erasured pool at 30% full. How will the cache tier manage it ?
[14:54] <circ-user-rwzoV> 2. Does it copy the whole file from scratch to the erasure-coded pool, or it has a smarter way to prevent that much I/O ?
[15:01] * elder_ (~elder@50.250.6.141) has joined #ceph
[15:05] * shaunm (~shaunm@208.102.161.229) has joined #ceph
[15:06] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) has joined #ceph
[15:12] * rogierm (~rogierm@a82-94-41-183.adsl.xs4all.nl) has joined #ceph
[15:13] * ira (~ira@nat-pool-bos-t.redhat.com) has joined #ceph
[15:15] * mhack (~mhack@nat-pool-bos-t.redhat.com) has joined #ceph
[15:18] * rogierm (~rogierm@a82-94-41-183.adsl.xs4all.nl) Quit (Remote host closed the connection)
[15:18] * rogierm_ (~rogierm@a82-94-41-183.adsl.xs4all.nl) has joined #ceph
[15:20] * elder_ (~elder@50.250.6.141) Quit (Quit: Leaving)
[15:22] * elder_ (~elder@50.250.6.141) has joined #ceph
[15:22] * linjan_ (~linjan@176.195.50.55) has joined #ceph
[15:23] * Grimhound (~FierceFor@4MJAAAZWN.tor-irc.dnsbl.oftc.net) Quit ()
[15:23] * nardial (~ls@dslb-088-072-094-077.088.072.pools.vodafone-ip.de) Quit (Quit: Leaving)
[15:24] <TheSov> morning!
[15:24] <rkeene> Lies.
[15:25] <TheSov> circ-user-rwzoV, you know ive never built a cache tier so i dont know but others here have if you wanna hang around
[15:25] <TheSov> rkeene, hello!
[15:25] * morse_ (~morse@supercomputing.univpm.it) Quit (Remote host closed the connection)
[15:27] * overclk (~vshankar@59.93.69.0) has joined #ceph
[15:27] * ira (~ira@nat-pool-bos-t.redhat.com) Quit (Remote host closed the connection)
[15:27] * jcsp (~jspray@82-71-16-249.dsl.in-addr.zen.co.uk) has joined #ceph
[15:27] * dyasny (~dyasny@dsl.198.58.158.134.ebox.ca) has joined #ceph
[15:27] <TheSov> the entirety of my new years and following weekend is a blur
[15:30] * squizzi (~squizzi@107.13.31.195) has joined #ceph
[15:30] * swami1 (~swami@49.44.57.238) Quit (Quit: Leaving.)
[15:31] * morse (~morse@supercomputing.univpm.it) has joined #ceph
[15:32] <DeMiNe0> I'm installing a new OSD host. It's main HDD for the OS is only 40gb and the machine has 24gb ram. Should I put a small 4gb swap, or not even bother with one? 8TB of total raw storage on this host.
[15:32] * rogierm_ (~rogierm@a82-94-41-183.adsl.xs4all.nl) Quit (Remote host closed the connection)
[15:33] <TheSov> you should always have a swap part, thats just good linux adminning.4gb should be enough
[15:33] * squizzi (~squizzi@107.13.31.195) Quit ()
[15:34] * squizzi (~squizzi@107.13.31.195) has joined #ceph
[15:36] <IcePic> or, when you have swapped out 1G, the machine will be so boring to use that it really isnt better at that point. Especially if the swap device is rather slow
[15:36] * rogierm (~rogierm@2001:985:1c56:1:417a:f845:3e6c:edaf) has joined #ceph
[15:38] <IcePic> "better than nothing" can still be "not good"
[15:38] <TheSov> in reality the swap space is used for zombie program memory and crap thats never recalled by from kernel memory again
[15:39] <TheSov> especially for systems that serve a single purpose like osd hosts
[15:39] * harold (~hamiller@71-94-227-123.dhcp.mdfd.or.charter.com) has joined #ceph
[15:40] <IcePic> I'm just leaning towards that if they have to use swap, then ram will not be used much for filesystem buffers or caching, and that would hurt more than doing the actual paging.
[15:40] * harold (~hamiller@71-94-227-123.dhcp.mdfd.or.charter.com) Quit ()
[15:41] <IcePic> but still, I would recommend having swap, if only to be able to make sure you can fsck large partitions at boot, in case it decides to need a lot of memory.
[15:41] <IcePic> in that case, its better to swap than to not be able to fsck at all
[15:42] <doppelgrau> or after a really goof ???fuck up??? of the cluster, the OSDs can need much more memory
[15:43] <doppelgrau> I was once happy that I had some swap after a whole cluster blackout with multiple disks dieing and some consequitve restarts => nearly all PGS were revovery and/or remapped
[15:44] * brad_mssw (~brad@66.129.88.50) has joined #ceph
[15:45] * garphy`aw is now known as garphy
[15:48] <circ-user-rwzoV> Guys, Does a cache tier for an erasure-coded pool copy the whole file from scratch to the erasure-coded pool, or it has a smarter way to prevent that much I/O ?
[15:50] <SamYaple> circ-user-rwzoV: on read it does that is my understanding (and real-world performance agrees with me)
[15:50] <SamYaple> circ-user-rwzoV: they are optimizing that, i saw some work on that for I and J releases
[15:50] * mattbenjamin (~mbenjamin@aa2.linuxbox.com) has joined #ceph
[15:50] <SamYaple> On Hammer the cache tier _can_ be useful, depending on your load type
[15:52] * yanzheng (~zhyan@182.139.23.32) Quit (Quit: This computer has gone to sleep)
[15:52] * dvanders (~dvanders@dvanders-pro.cern.ch) has joined #ceph
[15:53] * fghaas (~florian@91-119-120-46.dynamic.xdsl-line.inode.at) has joined #ceph
[15:53] * elder_ (~elder@50.250.6.141) Quit (Quit: Leaving)
[15:53] <circ-user-rwzoV> SamYaple: by real-world performances you mean it has low performances, right ?
[15:53] * rogierm (~rogierm@2001:985:1c56:1:417a:f845:3e6c:edaf) Quit (Remote host closed the connection)
[15:53] <circ-user-rwzoV> Just to clearify it
[15:53] * rmart04 (~rmart04@support.memset.com) has joined #ceph
[15:54] <SamYaple> circ-user-rwzoV: mostly yes. i have good success with it on a smaller cluster using fast ssd cache for openstack instances backed by spinning disks as the slow tier
[15:54] <SamYaple> but thats mostly due to the share data between all the instances
[15:55] <SamYaple> (the instances are all snapshots based on the same image)
[15:55] <circ-user-rwzoV> SamYaple: And when I append to a file, does it remove the old instance at the erasure-coded pool and rewrite it ?
[15:56] * rogierm_ (~rogierm@2001:985:1c56:1:4935:95bb:d7a0:1a60) has joined #ceph
[15:56] <SamYaple> i only have experince with the rbd+erasure-coding
[15:58] <circ-user-rwzoV> either I don't see why it matters, or you missed my point. If I write to a pool using librados, and append X times.
[15:59] <circ-user-rwzoV> the cache tier will copy to the erasure-coded pool, then delete and rewrite and so on ?
[15:59] <TheSov> i am aware that the LIO module for ceph is not yet ready, in the mean time, do I still use LIO for iscsi or should i go with something like tgt?
[15:59] * dneary (~dneary@nat-pool-bos-u.redhat.com) has joined #ceph
[16:01] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) Quit (Remote host closed the connection)
[16:01] <SamYaple> circ-user-rwzoV: when writing to the cachetier it won't immediately flush to the backing pool. but it will eventually and then it rewrites. rbds do not support partial-writes (part of the reason you can't rbd on an erasure-pool directly) so it does matter
[16:01] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) has joined #ceph
[16:01] <SamYaple> in the case of rbd you are rewriting the whole object, so that will of course be a full rewrite when it flushes back
[16:02] <circ-user-rwzoV> SamYaple: Got you.
[16:02] <circ-user-rwzoV> What do you do if your file is 100G then ?
[16:03] <SamYaple> see thats where you are losing me. Are you talking about rbd or cephfs?
[16:04] <circ-user-rwzoV> I'm looking for an option where I can write erasure, as optimal as I can. So I'm asking how do you deal with big-scale files
[16:04] * davidz (~davidz@2605:e000:1313:8003:489e:f5c2:6b41:d74) has joined #ceph
[16:05] <circ-user-rwzoV> -At one write-
[16:06] <circ-user-rwzoV> You said you have experience with rbd which doesn't have partial writes. So I meant, how do you write large files (100G+)
[16:06] <doppelgrau> circ-user-rwzoV: split the large file in smaler objects
[16:07] <SamYaple> yea for rbds circ-user-rwzoV they are split into (by default) 4MB objects
[16:07] <SamYaple> with parital writes maybe you can get that down even more. but youll want to split the object
[16:07] <SamYaple> don't store 100GB objects, thats a bad idea
[16:07] <TheSov> use --order to change the size
[16:07] <TheSov> 12 is 4k objects
[16:08] <circ-user-rwzoV> And when you want to read the "file", how do you ask for all the relevant objects ?
[16:08] <circ-user-rwzoV> What's connecting them together I mean
[16:08] <TheSov> the rbd client will do that
[16:08] <TheSov> the ceph client on the client system
[16:09] <TheSov> ceph is more a loose bundle of file servers than an actual cluster
[16:09] <TheSov> the monitors are the only portion that need constant communication
[16:09] <circ-user-rwzoV> Okay, I'm looking at it from a different place.
[16:09] <TheSov> the monitors provide clients a map of the data and some ip's and in effect the client becomes the "controller"
[16:10] <circ-user-rwzoV> My app already divides my file into small chunks. Do you have a solution that will make me write them into erasure coded pool without a cache tier ?
[16:10] <TheSov> i am confused, what are you trying to do, give me a 30,000 foot view
[16:11] <circ-user-rwzoV> Okay. I have video files. big ones
[16:11] * schmee (~simeon@41.78.128.114) has joined #ceph
[16:11] <TheSov> ok
[16:11] * schmee is now known as simeon
[16:12] <circ-user-rwzoV> My app take one video file, divides it into chunks, and send each chunk at a time to Ceph
[16:12] * steveeJ (~junky@HSI-KBW-149-172-252-139.hsi13.kabel-badenwuerttemberg.de) Quit (Ping timeout: 480 seconds)
[16:12] <circ-user-rwzoV> using librados, with the "append" command each time. (for a replicated pool)
[16:12] <TheSov> is dividing the file necessary?
[16:13] <TheSov> ok you are using librados, are you writing directly to a pool or are you using an RBD?
[16:13] <doppelgrau> circ-user-rwzoV: append is currently noch possible with erasure coded pools without a (small) cache tier
[16:13] <circ-user-rwzoV> Yeah, it's already exist and running with other clients without ceph, I just have to make it work with ceph
[16:13] * tsg (~tgohad@134.134.137.73) has joined #ceph
[16:14] <doppelgrau> circ-user-rwzoV: so if your application cann ???spool??? the file elsewhere and the pushes it to ceph you can avoid a cache pool
[16:14] <circ-user-rwzoV> TheSov : directly to a pool.
[16:14] <circ-user-rwzoV> doppelgrau: I know, that's where the problem is
[16:14] <circ-user-rwzoV> Using cache tier, I have too much IO
[16:15] <circ-user-rwzoV> I'm thinking about writing the whole file together
[16:15] <circ-user-rwzoV> But I can't see to find a way for that
[16:15] * whydidyoustealmynick (~shakamuny@c-67-180-191-38.hsd1.ca.comcast.net) Quit (Remote host closed the connection)
[16:15] <circ-user-rwzoV> Because chunks is a must
[16:15] <doppelgrau> circ-user-rwzoV: not even with some SSD?
[16:16] <doppelgrau> circ-user-rwzoV: rados gw or librados directly
[16:17] <circ-user-rwzoV> I'd still have to have much more storage than now. Today I have other RAID solution (not Ceph) which doesn't need a cache tier
[16:18] <circ-user-rwzoV> with the cache tier the storage costs are a much higher
[16:19] <doppelgrau> circ-user-rwzoV: the cache-pool can be quitesmall if you use it only sor ???assemling??? erasure coded objects, the SSDs only need a high wirte durability
[16:19] <SamYaple> doppelgrau: that will make it a big hit for performance though
[16:19] <SamYaple> double write _and_ reads
[16:19] <doppelgrau> only if the object is not in the cache
[16:20] * kefu (~kefu@114.92.107.250) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[16:20] <doppelgrau> I don???t know your use case: video surveilance?
[16:20] <doppelgrau> so usually lots of writes, reads only the most current and sometimes very old?
[16:21] <circ-user-rwzoV> doppelgrau: Yeah
[16:21] <doppelgrau> in that case most of the reads are comming popably from the cache pool => no double reads
[16:21] * kefu (~kefu@114.92.107.250) has joined #ceph
[16:22] <circ-user-rwzoV> surveilance: But twice the writes, and a lot more storage for keeping all that cache for read & write
[16:22] <doppelgrau> and double wirtes: no performance hit, the platters gets ???nice large objects??? => less seeking that can happen with appends when multiple streams hitting the same disk
[16:23] <doppelgrau> circ-user-rwzoV: ???all that cache??? could be three or four SSDs (but in that case I guess they age quite fast due to the amount of data written)
[16:24] * shaunm (~shaunm@208.102.161.229) Quit (Ping timeout: 480 seconds)
[16:25] <circ-user-rwzoV> doppelgrau: The storage is dozens of petabytes
[16:25] <circ-user-rwzoV> not a 3 or four SSD's
[16:25] * Be-El (~quassel@fb08-bcf-pc01.computational.bio.uni-giessen.de) Quit (Ping timeout: 480 seconds)
[16:25] * Be-El (~quassel@fb08-bcf-pc01.computational.bio.uni-giessen.de) has joined #ceph
[16:26] <doppelgrau> circ-user-rwzoV: and? The cache tier needs only be so large, that it can support the read/write IO
[16:28] <doppelgrau> circ-user-rwzoV: thats the idea behind a cache pool, you have a large ???cold??? storage that is efficent encoded, and a (way) smaller cache pool that is replicated for fast and easy data manipuation
[16:29] * olid1987 (~olid1982@p5484A288.dip0.t-ipconnect.de) has joined #ceph
[16:29] * shakamunyi (~shakamuny@c-67-180-191-38.hsd1.ca.comcast.net) has joined #ceph
[16:31] <circ-user-rwzoV> doppelgrau: At a given moment there are writes of ~5Gb/s. Let's say a standard file is 3GB, I'll need a lot of space. to handle it
[16:31] <SamYaple> doppelgrau: yea but writes are going to push out those reads, especially on a small cache tier
[16:31] * fouxm (~foucault@ks01.commit.ninja) Quit (Max SendQ exceeded)
[16:32] * fouxm (~foucault@ks01.commit.ninja) has joined #ceph
[16:35] * fouxm (~foucault@ks01.commit.ninja) Quit (Max SendQ exceeded)
[16:35] <Be-El> is anyone using nfs-ganesha with the cephfs-fsal? I'm currently encountering some problems with symlink targets...
[16:35] * fouxm (~foucault@ks01.commit.ninja) has joined #ceph
[16:35] * olid1986 (~olid1982@aftr-185-17-204-109.dynamic.mnet-online.de) Quit (Ping timeout: 480 seconds)
[16:35] <TheSov> whats cephfs-fsal?
[16:35] <TheSov> i am using ganesha with cephfs
[16:35] <TheSov> its in my experimental setup
[16:35] <SamYaple> hows that working TheSov? I think we talked about that before
[16:35] <SamYaple> the parallel nfs?
[16:36] <TheSov> i have a single cephfs setup on 4 systems
[16:36] <Be-El> TheSov: the fsal is the cephfs binding of ganesha
[16:36] <TheSov> those 4 systems have ganesha running nfs exports to vmware
[16:37] * zenpac (~zenpac3@66.55.33.66) has joined #ceph
[16:37] * enax (~enax@hq.ezit.hu) Quit (Ping timeout: 480 seconds)
[16:37] <TheSov> SamYaple, not parallel each nfs server is locked to its group of 2 vm hosts
[16:37] * amote (~amote@121.244.87.116) has joined #ceph
[16:38] <TheSov> and each group of 2 vm hosts has 2 nfs servers
[16:38] <TheSov> im trying to get a nfs setup with no SPOF
[16:39] <TheSov> but thats proving difficult with anything that doesnt have a ceph client
[16:40] <fghaas> Be-El: not sure if ndevos is around; he might know
[16:40] <Be-El> TheSov: which version of ganesha do you use? the 2.2.0 seems to broken in way too many places
[16:40] <TheSov> well crap now that i looked it up i may have done this wrong
[16:41] <TheSov> i mounted the cephfs locally and used ganesha to reexport
[16:41] <TheSov> but it appears ganesha has a ceph client module
[16:41] <TheSov> let me see
[16:42] <TheSov> this would explain why my file locks test arent working
[16:42] <TheSov> it is 2.2
[16:43] <Be-El> TheSov: that's the version i've trouble with...
[16:43] <TheSov> well i am not using the sal or whatever
[16:44] <Be-El> TheSov: symlinks are broken in the cephfs fsal, I get segfaults and SIGABRT and other "funny" thins
[16:44] <TheSov> im literally just re-exporting cephfs from the local system
[16:44] <Be-El> TheSov: ceph-fuse or kernel-based mount?
[16:44] <TheSov> fuse
[16:44] * linuxkidd (~linuxkidd@49.sub-70-209-96.myvzw.com) has joined #ceph
[16:44] <rkeene> Why not kernel cephfs ?
[16:44] <TheSov> rkeene, i never bothered to install it
[16:44] <rkeene> (Also, I'm experimenting with moving my OSDs to ZFS from XFS)
[16:44] <Be-El> TheSov: i had a similar setup before with kernel nfs as single nfs server. did not work at all with ceph-fuse
[16:44] * JCL (~JCL@ip68-224-244-110.lv.lv.cox.net) has joined #ceph
[16:44] <TheSov> i know its not the coolest but i like fuse
[16:45] * steveeJ (~junky@HSI-KBW-149-172-252-139.hsi13.kabel-badenwuerttemberg.de) has joined #ceph
[16:46] <rkeene> I can get behind FUSE with enough cache flags, but FUSE has a bug that bugs me for my FUSE-based app and as far as I can tell there's no workaround
[16:47] <TheSov> well i can tell you right now with ubuntu server 14.04 and ganesha 2.2 using cephfs fuse i can re-export just fine
[16:47] * JCL (~JCL@ip68-224-244-110.lv.lv.cox.net) Quit ()
[16:47] <Be-El> TheSov: and are clients able to edit files? I was not able to edit a text file with vi on client side and kernel nfs and ceph-fuse reexport on server side
[16:48] <TheSov> i dont know i connected it directly to vmware
[16:48] <TheSov> and at this point i am able to start vm's
[16:49] <ndevos> Be-El: if you have problems with the cephfs-fsal, you should probably report them to nfs-ganesha-devel@lists.sourceforge.net
[16:50] <ndevos> Be-El: and you would need to make the nfs-ganesha server high-available with pacemaker or such, relocating the ganesha service upon failure (no active-active option atm)
[16:51] <fghaas> thanks ndevos :)
[16:51] <Be-El> ndevos: there are currently to many problems with ganesha. I will report the errors, but disable ganesha and runs kernel based nfs
[16:52] <ndevos> np fghaas!
[16:53] <ndevos> Be-El: we're using it a lot in combination with Gluster, and things have been pretty stable, but that is 2.2+patches, I guess
[16:53] <Be-El> ndevos: does ganesha use system locales for filenames? the affected symlinks looks like this:mercy -> /ceph/biotools/share/celera_assembler/latest/bin/mercygger??:?
[16:53] <rkeene> I just ended up use cephfs on all the clients, it's been working out a lot better (before I was using a filesystem on a single RBD exported out, rather than exporting out cephfs)
[16:53] * JCL (~JCL@ip68-224-244-110.lv.lv.cox.net) has joined #ceph
[16:53] <fghaas> Be-El: are you sure that's not just your LC_CONSOLE setting?
[16:54] <ndevos> Be-El: it should not, but I dont know how libcephfs (is that the name?) handles things, Ganesha should just pass filenames on without changing them
[16:54] * Steppy (~dleeuw@143.121.192.183) has joined #ceph
[16:55] <Be-El> fghaas: locale is set to en_US for almost all environment settings
[16:55] <Be-El> fghaas: and the filename should end at /mercy
[16:56] <ndevos> Be-El: there is also #ganesha on Freenode, some of the guys there would know more about fsal-cephfs than me
[16:56] * moore (~moore@64.202.160.88) has joined #ceph
[16:56] <Be-El> ndevos: ah, good hint...i'll try there first
[16:57] <rkeene> I almost wrote an NFS server that just talked to librbd and exported RBDs rather than files
[16:57] <TheSov> rkeene, how would that work? doesnt it have to provide file level access?
[16:57] * tsg (~tgohad@134.134.137.73) Quit (Remote host closed the connection)
[16:57] * tsg (~tgohad@134.134.137.73) has joined #ceph
[16:58] <rkeene> TheSov, Each RBD image would be a file
[16:58] <TheSov> but then how would that work on the client side?
[16:58] <rkeene> So <mountpoint>/<pool>/<image>
[16:59] <TheSov> like on the client side i mount the nfs right? what would it look like if i did LS?
[16:59] <Steppy> I per accident killed my (only) mon and now it claims that there is no /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000002... Is there any way to fix this?
[16:59] <rkeene> What do you mean ? Pretty much exactly as it sounds, say you have the image "rbd/test", you'd have /mountpoint/rbd/test that when you issue a READ3/WRITE3 would operate on
[17:00] <TheSov> Steppy, you just stepped into a lot of shit
[17:00] * wushudoin (~wushudoin@2601:646:8201:7769:2ab2:bdff:fe0b:a6ee) has joined #ceph
[17:00] <Steppy> TheSov :(
[17:00] <Gugge-47527> rkeene: so basically as usefull as rbd-fuse ... and only usefull on clients without native rbd support?
[17:00] <TheSov> you basically just walked in the door and said the map of my data is corrupt
[17:01] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) Quit (Remote host closed the connection)
[17:01] * thomnico (~thomnico@2a01:e35:8b41:120:48c4:8169:429f:b1ce) Quit (Quit: Ex-Chat)
[17:01] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) has joined #ceph
[17:01] <TheSov> and thats why you never have only 1 monitor
[17:02] <Steppy> TheSov: Lucky enough this is a test env, with no crucial data on it...
[17:03] <Steppy> I was just wondering how to get back running from this situation, if at all possible.
[17:03] <Steppy> Else I wipe, have a cup of coffee and start again.
[17:03] <rkeene> Gugge-47527, Right, like VMware ESXi hosts -- it'd just be a thin bridge between NFS->librbd
[17:04] <Gugge-47527> but how would an esx host use that for anything?
[17:04] <rkeene> Gugge-47527, ESXi can talk NFS
[17:05] <Gugge-47527> yes, but it expect a nfs share where it can place files
[17:05] <Gugge-47527> not an nfs share with files it has to use as blockdevices
[17:05] * olid1988 (~olid1982@aftr-185-17-204-109.dynamic.mnet-online.de) has joined #ceph
[17:06] * mhack (~mhack@nat-pool-bos-t.redhat.com) Quit (Ping timeout: 480 seconds)
[17:06] <rkeene> Gugge-47527, Right, for some files it would have a different path where it stored them in another filesystem -- filename pattern based matching, so the .vmx and .log files would go to another filesystem while the .vmdk's paths would be rewritten to images
[17:06] * mtb` (~mtb`@157.130.171.46) has joined #ceph
[17:07] * analbeard (~shw@support.memset.com) Quit (Quit: Leaving.)
[17:08] * circ-user-rwzoV (~circuser-@sessfw99-sesbfw99-91.ericsson.net) Quit (Remote host closed the connection)
[17:09] * Tonux (~Ralth@7V7AAB2ZS.tor-irc.dnsbl.oftc.net) has joined #ceph
[17:10] * olid1987 (~olid1982@p5484A288.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[17:10] * RMar04 (~RMar04@support.memset.com) Quit (Ping timeout: 480 seconds)
[17:11] * rmart04 (~rmart04@support.memset.com) Quit (Ping timeout: 480 seconds)
[17:11] * yankcrime (~yankcrime@185.43.216.241) Quit (Quit: Textual IRC Client: www.textualapp.com)
[17:13] * mtanski_ (~mtanski@65.244.82.98) has joined #ceph
[17:13] * mtanski (~mtanski@65.244.82.98) Quit (Read error: Connection reset by peer)
[17:13] * mtanski_ is now known as mtanski
[17:13] * cholcombe (~chris@c-73-180-29-35.hsd1.or.comcast.net) has joined #ceph
[17:14] * yankcrime (~yankcrime@185.43.216.241) has joined #ceph
[17:16] <Be-El> TheSov: so reexporting ceph-fuse mount points work with ganesha in your setup?
[17:17] * mhack (~mhack@nat-pool-bos-t.redhat.com) has joined #ceph
[17:18] * TMM (~hp@sams-office-nat.tomtomgroup.com) Quit (Quit: Ex-Chat)
[17:22] * amote (~amote@121.244.87.116) Quit (Ping timeout: 480 seconds)
[17:23] * shaunm (~shaunm@cpe-74-132-70-216.kya.res.rr.com) has joined #ceph
[17:24] * Steppy (~dleeuw@143.121.192.183) Quit (Quit: Leaving.)
[17:26] * jdillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) has joined #ceph
[17:26] * zigo_ is now known as zigo
[17:27] * Eduardo__ (~Eduardo@bl4-180-165.dsl.telepac.pt) has joined #ceph
[17:30] * nardial (~ls@dslb-088-072-094-077.088.072.pools.vodafone-ip.de) has joined #ceph
[17:31] * joshd1 (~jdurgin@68-119-140-18.dhcp.ahvl.nc.charter.com) has joined #ceph
[17:31] * dgbaley27 (~matt@75.148.118.217) has joined #ceph
[17:33] * jwilkins (~jowilkin@2601:644:4000:97c0:ea2a:eaff:fe08:3f1d) has joined #ceph
[17:34] * Eduardo_ (~Eduardo@bl5-1-253.dsl.telepac.pt) Quit (Ping timeout: 480 seconds)
[17:35] * elder_ (~elder@50.250.13.173) has joined #ceph
[17:37] * kefu is now known as kefu|afk
[17:37] * nardial (~ls@dslb-088-072-094-077.088.072.pools.vodafone-ip.de) Quit (Quit: Leaving)
[17:39] * Tonux (~Ralth@7V7AAB2ZS.tor-irc.dnsbl.oftc.net) Quit ()
[17:41] * lkoranda (~lkoranda@nat-pool-brq-t.redhat.com) has joined #ceph
[17:41] * zaitcev (~zaitcev@c-50-130-189-82.hsd1.nm.comcast.net) has joined #ceph
[17:41] <TheSov> Be-El, yes
[17:42] * elder_ (~elder@50.250.13.173) Quit (Quit: Leaving)
[17:42] * treenerd (~treenerd@85.193.140.98) Quit (Ping timeout: 480 seconds)
[17:42] * reed (~reed@75-101-54-18.dsl.static.fusionbroadband.com) has joined #ceph
[17:43] * mhack is now known as mhack|lunch
[17:44] * rogierm_ (~rogierm@2001:985:1c56:1:4935:95bb:d7a0:1a60) Quit (Remote host closed the connection)
[17:44] * _s1gma (~Grimmer@6YRAABVTO.tor-irc.dnsbl.oftc.net) has joined #ceph
[17:44] * rogierm_ (~rogierm@a82-94-41-183.adsl.xs4all.nl) has joined #ceph
[17:44] * ibravo (~ibravo@72.83.69.64) has joined #ceph
[17:46] * wjw-freebsd (~wjw@vpn.ecoracks.nl) Quit (Ping timeout: 480 seconds)
[17:50] * kefu|afk (~kefu@114.92.107.250) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[17:51] * barra204 (~shakamuny@c-67-180-191-38.hsd1.ca.comcast.net) has joined #ceph
[17:51] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) Quit (Remote host closed the connection)
[17:52] * jclm (~jclm@ip68-224-244-110.lv.lv.cox.net) has joined #ceph
[17:52] * lkoranda_ (~lkoranda@nat-pool-brq-t.redhat.com) has joined #ceph
[17:54] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[17:55] * shakamunyi (~shakamuny@c-67-180-191-38.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[17:57] * lkoranda (~lkoranda@nat-pool-brq-t.redhat.com) Quit (Ping timeout: 480 seconds)
[17:57] * lkoranda_ is now known as lkoranda
[17:58] * chasmo77 (~chas77@158.183-62-69.ftth.swbr.surewest.net) Quit (Quit: It's just that easy)
[17:58] * alram (~alram@206.169.83.146) has joined #ceph
[18:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Remote host closed the connection)
[18:01] <Freeaqingme> I understand it's ready when it's ready, but what kind of timeline would be realistic for cephfs to be considered stable?
[18:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[18:04] * yankcrime (~yankcrime@185.43.216.241) Quit (Quit: Textual IRC Client: www.textualapp.com)
[18:06] * Steppy (~dennis@a80-100-8-179.adsl.xs4all.nl) has joined #ceph
[18:06] * visbits (~textual@8.29.138.28) has joined #ceph
[18:06] <visbits> i highly advise not doubling your pg count from 2048 to 4096 on 482TB of data in production
[18:07] <zao> Sounds like advice with a fun story behind it.
[18:07] * squizzi (~squizzi@107.13.31.195) Quit (Ping timeout: 480 seconds)
[18:07] <jcsp> Freeaqingme: The Jewel release will have the new scrub/fsck functionality, so you should plan around using that (this spring). No announced date for official support though.
[18:08] <visbits> well lets just say, i started it last night around 9pm, its now noon and were about 27% with the new pgs
[18:08] <visbits> with 275 osds
[18:08] <Freeaqingme> jcsp: cool. tnx
[18:12] * dgbaley27 (~matt@75.148.118.217) Quit (Quit: Leaving.)
[18:12] <Be-El> yeah, that's really good news
[18:14] * _s1gma (~Grimmer@6YRAABVTO.tor-irc.dnsbl.oftc.net) Quit ()
[18:14] * dugravot6 (~dugravot6@dn-infra-04.lionnois.univ-lorraine.fr) Quit (Quit: Leaving.)
[18:16] * georgem (~Adium@206.108.127.16) has joined #ceph
[18:24] * overclk (~vshankar@59.93.69.0) Quit (Quit: Zzzz...)
[18:24] * treenerd (~treenerd@178.165.128.119.wireless.dyn.drei.com) has joined #ceph
[18:26] * dgurtner (~dgurtner@178.197.231.6) Quit (Ping timeout: 480 seconds)
[18:28] * Kupo1 (~tyler.wil@23.111.254.159) has joined #ceph
[18:39] * jordanP (~jordan@pas38-2-82-67-72-49.fbx.proxad.net) Quit (Quit: Leaving)
[18:41] * mykola (~Mikolaj@91.225.201.213) has joined #ceph
[18:44] * squizzi (~squizzi@107.13.31.195) has joined #ceph
[18:45] * lcurtis (~lcurtis@47.19.105.250) has joined #ceph
[18:46] * InIMoeK (~InIMoeK@95.170.93.16) Quit (Ping timeout: 480 seconds)
[18:59] * debian112 (~bcolbert@24.126.201.64) has joined #ceph
[19:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Remote host closed the connection)
[19:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[19:01] <georgem> is there a way to re-weight up an OSD?
[19:01] <TheSov> yes
[19:02] <TheSov> use the normal command and then use a bigger number
[19:02] <georgem> it doesn't accept anything larger than 1
[19:03] <TheSov> in my cluster though the general rule is that the weight = number of TB of the disk
[19:03] <TheSov> what do you mean it wont accept anything larger than 1
[19:03] <georgem> I'm not talking about the crush weight
[19:03] <TheSov> i know...
[19:04] * danieagle (~Daniel@179.111.172.42) has joined #ceph
[19:04] <georgem> osd reweight <int[0-]> <float[0.0-1.0]> : reweight osd to 0.0 < <weight> < 1.0
[19:04] <TheSov> run ceph osd tree
[19:04] <TheSov> and sned output
[19:04] <TheSov> also what version of ceph are you running
[19:06] <Be-El> georgem: you cannot set an osd weight larger than 1.0
[19:06] <Be-El> georgem: you need to readjust the other weights in a relativ way
[19:06] <georgem> TheSov: I'm running Hammer and I already lowered the weight of some OSDs but the data just moves on other drives in same server or rack and very little goes to the bottom-loaded OSDs
[19:06] <Be-El> georgem: or use reweight-by-utilization
[19:06] <georgem> ok, so let me give more details and see how impacting this would be
[19:07] <TheSov> so why does adding an OSD by default have some large weight if you can never go back to that number
[19:08] * bene_in_mtg (~bene@nat-pool-bos-t.redhat.com) has joined #ceph
[19:08] * dmick (~dmick@206.169.83.146) has joined #ceph
[19:10] * Be-El (~quassel@fb08-bcf-pc01.computational.bio.uni-giessen.de) Quit (Remote host closed the connection)
[19:10] * angdraug (~angdraug@c-69-181-140-42.hsd1.ca.comcast.net) has joined #ceph
[19:10] * yuriw (~Adium@2601:645:4380:112c:c9c3:216b:8a5a:217b) Quit (Ping timeout: 480 seconds)
[19:12] * yuriw (~Adium@2601:645:4380:112c:154b:d781:1f27:2a5a) has joined #ceph
[19:13] * yuriw (~Adium@2601:645:4380:112c:154b:d781:1f27:2a5a) Quit ()
[19:15] * ngoswami (~ngoswami@121.244.87.116) Quit (Quit: Leaving)
[19:16] * ade (~abradshaw@dslb-092-078-128-021.092.078.pools.vodafone-ip.de) Quit (Quit: Too sexy for his shirt)
[19:18] * pabluk is now known as pabluk_
[19:19] * togdon (~togdon@c-76-115-1-100.hsd1.or.comcast.net) has joined #ceph
[19:19] * joshd1 (~jdurgin@68-119-140-18.dhcp.ahvl.nc.charter.com) Quit (Quit: Leaving.)
[19:19] * togdon (~togdon@c-76-115-1-100.hsd1.or.comcast.net) Quit ()
[19:20] * DanFoster (~Daniel@2a00:1ee0:3:1337:613d:7a75:fb8a:b1b) Quit (Quit: Leaving)
[19:20] <georgem> I have two racks, one rack with five nodes and 180 OSD and the second rack with seven nodes and 252 OSDs; the crush rule puts two replicas in one rack and one replica in the other, so I'm limited by 85% of the data the smallest rack can store, and 85% of the the most full OSD in that rack
[19:20] * vasu (~vasu@c-73-231-60-138.hsd1.ca.comcast.net) has joined #ceph
[19:21] * Etki (~smuxi@89-156-86-143.rev.numericable.fr) has joined #ceph
[19:22] * daviddcc (~dcasier@LAubervilliers-656-1-16-160.w217-128.abo.wanadoo.fr) Quit (Ping timeout: 480 seconds)
[19:22] <georgem> until now I change the weight using "ceph osd reweight OSD_ID 0.8" which limits the IO impact, is "reweight-by-utilization" going to create an I/O storm?
[19:25] * yuriw (~Adium@2601:645:4380:112c:7d2e:fa22:e54a:67e9) has joined #ceph
[19:25] * yuriw (~Adium@2601:645:4380:112c:7d2e:fa22:e54a:67e9) Quit (Remote host closed the connection)
[19:25] * yuriw (~Adium@2601:645:4380:112c:7d2e:fa22:e54a:67e9) has joined #ceph
[19:31] <TheSov> man thats like the number 1 complaint i hear about the cluster slowing down due to rebalancing, my philosophy on it is that, thats a normal operation for the cluster and people should expect it and live with it. if the cluster needed to be super fast all the time with 0 delays then you should get a flash DAS for those systems
[19:32] <TheSov> * not all systems may apply!
[19:33] <georgem> I agree, but I should also understand the impact before running it
[19:34] * treenerd (~treenerd@178.165.128.119.wireless.dyn.drei.com) Quit (Ping timeout: 480 seconds)
[19:35] * rendar_ (~I@87.19.183.44) has joined #ceph
[19:35] * dmick (~dmick@206.169.83.146) has left #ceph
[19:35] * derjohn_mobi (~aj@2001:6f8:1337:0:612a:ee5f:ae4:64aa) Quit (Ping timeout: 480 seconds)
[19:36] * rogierm_ (~rogierm@a82-94-41-183.adsl.xs4all.nl) Quit (Remote host closed the connection)
[19:38] * i_m1 (~ivan.miro@88.206.113.199) has joined #ceph
[19:39] * rendar (~I@87.19.183.44) Quit (Ping timeout: 480 seconds)
[19:40] * kaisan (~kai@zaphod.xs4all.nl) has joined #ceph
[19:45] * i_m (~ivan.miro@88.206.113.199) Quit (Ping timeout: 480 seconds)
[19:46] * i_m1 (~ivan.miro@88.206.113.199) Quit (Ping timeout: 480 seconds)
[19:46] * enax (~enax@94-21-125-58.pool.digikabel.hu) has joined #ceph
[19:46] * enax (~enax@94-21-125-58.pool.digikabel.hu) has left #ceph
[19:47] * dgbaley27 (~matt@75.148.118.217) has joined #ceph
[19:50] * jclm1 (~jclm@155.94.227.67) has joined #ceph
[19:51] * thansen (~thansen@17.253.sfcn.org) has joined #ceph
[19:53] <visbits> TheSov rebalancing barely impacts a properly setup network, now increasing pgs on a 400T pool.. that fucks shit up lol :\
[19:53] * alram_ (~alram@206.169.83.146) has joined #ceph
[19:55] * angdraug (~angdraug@c-69-181-140-42.hsd1.ca.comcast.net) Quit (Quit: Leaving)
[19:55] * jclm (~jclm@ip68-224-244-110.lv.lv.cox.net) Quit (Ping timeout: 480 seconds)
[20:00] * alram (~alram@206.169.83.146) Quit (Ping timeout: 480 seconds)
[20:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Remote host closed the connection)
[20:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[20:02] <SamYaple> visbits: properly setup being key
[20:02] <SamYaple> most setups rebalancing saturates the network and slows it down to a crawl
[20:06] <TheSov> your supposed to have a private network for that
[20:08] * derjohn_mobi (~aj@x590ceea9.dyn.telefonica.de) has joined #ceph
[20:09] <visbits> im saturating 40g atm
[20:09] <visbits> lol
[20:12] <TheSov> visbits, im working on smaller scale deployments, im trying to even build an appliance
[20:12] <visbits> yeah my ceph is 1200T 275 osds
[20:13] <TheSov> let me guess, EC with a cache tier
[20:16] * fghaas (~florian@91-119-120-46.dynamic.xdsl-line.inode.at) Quit (Ping timeout: 480 seconds)
[20:16] <visbits> wat
[20:18] <TheSov> are you running your cluster as EC pools with a cache tiers?
[20:19] <TheSov> or you doing straight rust with SSD journals?
[20:28] <visbits> its rust with no journal
[20:28] <visbits> we've not found that journal improves performance at all at this scale
[20:29] * rotbeard (~redbeard@185.32.80.238) Quit (Quit: Leaving)
[20:29] <visbits> until cephs software latency comes down more with the next release that includes the sandisk fixes it wont help
[20:29] <visbits> i think its a 30ms write with ceph? a spindle writes in 12-15
[20:30] <TheSov> well its difficult to do when you got all that software in the way
[20:30] * jrankin (~jrankin@d53-64-170-236.nap.wideopenwest.com) Quit (Quit: Leaving)
[20:35] * fghaas (~florian@91-119-120-46.dynamic.xdsl-line.inode.at) has joined #ceph
[20:38] * Grum (~EdGruberm@185.60.144.31) has joined #ceph
[20:38] * wer (~wer@216.197.66.226) Quit (Remote host closed the connection)
[20:38] * wer (~wer@216.197.66.226) has joined #ceph
[20:39] * nigwil (~Oz@li1416-21.members.linode.com) has joined #ceph
[20:48] * InIMoeK (~InIMoeK@029-226-128-083.dynamic.caiway.nl) has joined #ceph
[20:59] * Steppy (~dennis@a80-100-8-179.adsl.xs4all.nl) has left #ceph
[21:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Remote host closed the connection)
[21:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[21:05] * simeon (~simeon@41.78.128.114) has left #ceph
[21:08] * Grum (~EdGruberm@84ZAAA2WU.tor-irc.dnsbl.oftc.net) Quit ()
[21:10] * DV (~veillard@2001:41d0:a:f29f::1) Quit (Remote host closed the connection)
[21:10] * DV (~veillard@2001:41d0:a:f29f::1) has joined #ceph
[21:10] * derjohn_mobi (~aj@x590ceea9.dyn.telefonica.de) Quit (Ping timeout: 480 seconds)
[21:11] * mtb` (~mtb`@157.130.171.46) Quit (Quit: Textual IRC Client: www.textualapp.com)
[21:12] * Icey (~Icey@pool-74-109-7-163.phlapa.fios.verizon.net) has joined #ceph
[21:18] * mtb` (~mtb`@157.130.171.46) has joined #ceph
[21:18] * mortn (~mortn@217-215-219-69-no229.tbcn.telia.com) has joined #ceph
[21:21] * rogierm (~rogierm@2001:985:1c56:1:59d5:544d:c837:f287) has joined #ceph
[21:26] * rogierm (~rogierm@2001:985:1c56:1:59d5:544d:c837:f287) Quit (Remote host closed the connection)
[21:30] * linuxkidd (~linuxkidd@49.sub-70-209-96.myvzw.com) Quit (Ping timeout: 480 seconds)
[21:31] * linuxkidd (~linuxkidd@49.sub-70-209-96.myvzw.com) has joined #ceph
[21:33] * johnavp1989 (~jpetrini@8.39.115.8) has joined #ceph
[21:40] * wjw-freebsd (~wjw@smtp.digiware.nl) has joined #ceph
[21:44] * luigiman (~Hejt@tor3.digineo.de) has joined #ceph
[21:45] * ira (~ira@24.34.255.34) has joined #ceph
[21:55] * rogierm_ (~rogierm@a82-94-41-183.adsl.xs4all.nl) has joined #ceph
[21:56] * owasserm (~owasserm@2001:984:d3f7:1:5ec5:d4ff:fee0:f6dc) Quit (Read error: Permission denied)
[21:57] * owasserm (~owasserm@2001:984:d3f7:1:5ec5:d4ff:fee0:f6dc) has joined #ceph
[21:59] * daviddcc (~dcasier@80.12.55.237) has joined #ceph
[22:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Remote host closed the connection)
[22:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[22:12] * mykola (~Mikolaj@91.225.201.213) Quit (Quit: away)
[22:14] * luigiman (~Hejt@6YRAABVXP.tor-irc.dnsbl.oftc.net) Quit ()
[22:15] * derjohn_mob (~aj@tmo-112-174.customers.d1-online.com) has joined #ceph
[22:15] * wjw-freebsd (~wjw@smtp.digiware.nl) Quit (Ping timeout: 480 seconds)
[22:15] * yguang11 (~yguang11@2001:4998:effd:600:99e1:b2f:e2cf:48c0) has joined #ceph
[22:17] * alram_ (~alram@206.169.83.146) Quit (Ping timeout: 480 seconds)
[22:19] * geli1 (~geli@geli-2015.its.utas.edu.au) Quit (Ping timeout: 480 seconds)
[22:21] * geli (~geli@geli-2015.its.utas.edu.au) has joined #ceph
[22:24] * InIMoeK (~InIMoeK@029-226-128-083.dynamic.caiway.nl) Quit (Ping timeout: 480 seconds)
[22:25] * wjw-freebsd (~wjw@smtp.digiware.nl) has joined #ceph
[22:27] * dusti (~drdanick@76GAAA0GG.tor-irc.dnsbl.oftc.net) has joined #ceph
[22:36] * dgbaley27 (~matt@75.148.118.217) Quit (Quit: Leaving.)
[22:39] * alram (~alram@cpe-172-250-2-46.socal.res.rr.com) has joined #ceph
[22:41] * marrusl (~mark@209-150-46-243.c3-0.wsd-ubr2.qens-wsd.ny.cable.rcn.com) has joined #ceph
[22:45] * rogierm_ (~rogierm@a82-94-41-183.adsl.xs4all.nl) Quit (Remote host closed the connection)
[22:48] * rogierm (~rogierm@2001:985:1c56:1:4cfe:de67:f103:f644) has joined #ceph
[22:57] * dusti (~drdanick@76GAAA0GG.tor-irc.dnsbl.oftc.net) Quit ()
[22:57] * nardial (~ls@dslb-088-072-094-077.088.072.pools.vodafone-ip.de) has joined #ceph
[23:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) Quit (Remote host closed the connection)
[23:01] * haomaiwang (~haomaiwan@li745-113.members.linode.com) has joined #ceph
[23:03] * danieagle (~Daniel@179.111.172.42) Quit (Quit: Obrigado por Tudo! :-) inte+ :-))
[23:03] * cloudm2 (uid37542@id-37542.tooting.irccloud.com) has joined #ceph
[23:07] * DV (~veillard@2001:41d0:a:f29f::1) Quit (Remote host closed the connection)
[23:07] * DV (~veillard@2001:41d0:a:f29f::1) has joined #ceph
[23:12] * georgem (~Adium@206.108.127.16) Quit (Quit: Leaving.)
[23:14] * mhack|lunch is now known as mhack|afk
[23:14] * olid1989 (~olid1982@p5484AF0F.dip0.t-ipconnect.de) has joined #ceph
[23:18] * olid1988 (~olid1982@aftr-185-17-204-109.dynamic.mnet-online.de) Quit (Ping timeout: 480 seconds)
[23:20] * rogierm (~rogierm@2001:985:1c56:1:4cfe:de67:f103:f644) Quit (Remote host closed the connection)
[23:27] * olid19810 (~olid1982@aftr-185-17-204-109.dynamic.mnet-online.de) has joined #ceph
[23:28] * nardial (~ls@dslb-088-072-094-077.088.072.pools.vodafone-ip.de) Quit (Quit: Leaving)
[23:30] * angdraug (~angdraug@64.124.158.100) has joined #ceph
[23:31] * marrusl (~mark@209-150-46-243.c3-0.wsd-ubr2.qens-wsd.ny.cable.rcn.com) Quit (Quit: bye!)
[23:33] * olid1989 (~olid1982@p5484AF0F.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[23:42] * Moriarty (~Fapiko@185.60.144.31) has joined #ceph
[23:44] * johnavp1989 (~jpetrini@8.39.115.8) Quit (Ping timeout: 480 seconds)
[23:45] * brad_mssw (~brad@66.129.88.50) Quit (Quit: Leaving)
[23:46] * shaunm (~shaunm@cpe-74-132-70-216.kya.res.rr.com) Quit (Ping timeout: 480 seconds)
[23:48] * LDA (~DM@host217-114-156-249.pppoe.mark-itt.net) Quit (Quit: Nettalk6 - www.ntalk.de)
[23:51] * chasmo77 (~chas77@158.183-62-69.ftth.swbr.surewest.net) has joined #ceph
[23:52] * mortn (~mortn@217-215-219-69-no229.tbcn.telia.com) Quit (Quit: Leaving)

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.