#ceph IRC Log

Index

IRC Log for 2016-09-28

Timestamps are in GMT/BST.

[0:00] * KindOne (kindone@h26.226.28.71.dynamic.ip.windstream.net) has joined #ceph
[0:02] * jclm (~jclm@ip68-96-196-245.lv.lv.cox.net) has joined #ceph
[0:02] * jclm (~jclm@ip68-96-196-245.lv.lv.cox.net) Quit (Remote host closed the connection)
[0:04] * vbellur (~vijay@71.234.224.255) has joined #ceph
[0:09] * vata (~vata@207.96.182.162) Quit (Quit: Leaving.)
[0:11] * ChuckMe_ (~ChuckMe@MTRLPQ42-1176054809.sdsl.bell.ca) Quit (Quit: ChuckMe)
[0:11] * xinli (~charleyst@32.97.110.56) Quit (Ping timeout: 480 seconds)
[0:18] * MK_FG (~MK_FG@00018720.user.oftc.net) has joined #ceph
[0:20] * blizzow (~jburns@172.56.15.63) has joined #ceph
[0:25] * pdrakeweb (~pdrakeweb@cpe-71-74-153-111.neo.res.rr.com) Quit (Ping timeout: 480 seconds)
[0:38] * blizzow (~jburns@172.56.15.63) Quit (Ping timeout: 480 seconds)
[0:40] <axion_joey> Hey Everyone, I have a question about performace
[0:40] <axion_joey> performance
[0:40] <axion_joey> I set up 2 clusters
[0:40] <axion_joey> one has 4 servers. each server has 6 platter drives in it
[0:41] <axion_joey> the other cluster has 8 servers. each server has 4 SSD's in it
[0:41] <axion_joey> When I run rados bench agains the platter cluster it shows an average bandwidth of 111MB/s
[0:42] <axion_joey> the ssd cluster shows an average bandwidth of 165MB/s
[0:42] <axion_joey> shouldn't I see a much larger performance improvement by using SSD's and having double the number of hosts?
[0:44] * stiopa (~stiopa@cpc73832-dals21-2-0-cust453.20-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[0:45] * pdrakeweb (~pdrakeweb@oh-76-5-101-140.dhcp.embarqhsd.net) has joined #ceph
[0:56] * vata (~vata@96.127.202.136) has joined #ceph
[0:56] * pdrakewe_ (~pdrakeweb@cpe-71-74-153-111.neo.res.rr.com) has joined #ceph
[0:58] * pdrakeweb (~pdrakeweb@oh-76-5-101-140.dhcp.embarqhsd.net) Quit (Ping timeout: 480 seconds)
[1:03] * pdrakewe_ (~pdrakeweb@cpe-71-74-153-111.neo.res.rr.com) Quit (Read error: No route to host)
[1:03] * pdrakeweb (~pdrakeweb@cpe-71-74-153-111.neo.res.rr.com) has joined #ceph
[1:10] * ronrib (~boswortr@45.32.242.135) Quit (Quit: Leaving.)
[1:10] * sudocat (~dibarra@192.185.1.20) Quit (Read error: Connection reset by peer)
[1:13] <doppelgrau> axion_joey: what???s your network setup?
[1:14] <axion_joey> 2 10 gig ports in each OSD host configured as LACP
[1:14] <doppelgrau> axion_joey: rados bench uses large (4MB) objects, so it???s more or less sequential IO => platters aren???t that bad
[1:14] * vicente (~vicente@1-161-184-59.dynamic.hinet.net) has joined #ceph
[1:14] <axion_joey> got it
[1:15] <axion_joey> I saw a presentation that said that ceph write performance around 64% of the combined throughput of the underlying disks
[1:15] <axion_joey> so I was expecting higher performance
[1:16] * debian112 (~bcolbert@c-73-184-103-26.hsd1.ga.comcast.net) Quit (Ping timeout: 480 seconds)
[1:16] <axion_joey> it's still more than what we need, but now we're just questioning if it's worth it to pay for SSD's vs platters
[1:17] <doppelgrau> and 24 OSDs (platter) vs. 32 OSDs (SSDs) so there is almost a linear trend with your performance, but you???re right, SSDs should be faster
[1:17] <axion_joey> I thought ceph performance was more determined by the number of hosts than the number of drives
[1:17] <doppelgrau> but overall performance is still not really good IMHO
[1:17] <axion_joey> that's why we did 8 ssd hosts instead of 4
[1:18] <axion_joey> agreed
[1:18] <axion_joey> are there any performance tuning tips
[1:18] <axion_joey> I feel like we're leaving a lot on the table
[1:18] * vasu (~vasu@38.140.108.5) Quit (Ping timeout: 480 seconds)
[1:18] <doppelgrau> as long bandwidth and cpu are not the limits usually the drives are (but with good SSDs you should usually get far more performance that spinning disks)
[1:19] <axion_joey> the drives aren't really high end. They're Sandisk Extreme Pros
[1:19] <axion_joey> still they should be much faster than traditional
[1:20] <doppelgrau> axion_joey: try a read benchmark
[1:21] <axion_joey> on it
[1:22] * vicente (~vicente@1-161-184-59.dynamic.hinet.net) Quit (Ping timeout: 480 seconds)
[1:23] <doppelgrau> axion_joey: either drop caches on the osd-server or benachmark with too many data for the memory :)
[1:24] <doppelgrau> axion_joey: else the read-benchmark might be ???slightly??? skewed
[1:24] <axion_joey> good call
[1:25] <axion_joey> thanks
[1:25] <doppelgrau> but if the ssds are almost the same speed again, I???d double check the network OS
[1:26] <axion_joey> ok
[1:26] <axion_joey> I ran the read test on the platters first
[1:26] <axion_joey> 1.5GB/s
[1:26] <doppelgrau> but if the SSDs are faster, you might take a look, if these SSDs are good for the ceph write pattern, some are getting very slow with sync writes
[1:26] <axion_joey> SSD read is same as platter read
[1:27] <axion_joey> probably hitting the network limit of the server running the benchmark
[1:27] <doppelgrau> axion_joey: reduce thread count -t 1
[1:27] * debian112 (~bcolbert@c-73-184-103-26.hsd1.ga.comcast.net) has joined #ceph
[1:28] <axion_joey> 411MB/s avg random read on the SSD's
[1:28] <doppelgrau> with droped caches HDDs should return something about 100 MB/s
[1:28] <doppelgrau> (less with random)
[1:29] * jclm (~jclm@ip68-96-196-245.lv.lv.cox.net) has joined #ceph
[1:30] <doppelgrau> (1 thread = one concurent read and platters usually do not read much faster than 100 MB/s)
[1:30] <axion_joey> hard drives are also showing 400MB rand read with 1 thread
[1:30] <doppelgrau> did you drop the caches on the osd-servers or only on the benchmark-host?
[1:31] <axion_joey> dang it
[1:31] <doppelgrau> I guess the data is just in memory
[1:31] <axion_joey> i only did it on the benchmark host
[1:32] <axion_joey> are there any recommendations for SSD's that provide good performance that aren't as expensive as the Intel drives?
[1:33] <doppelgrau> axion_joey: the sm863(?) from samsung are in use for journals (platter for storage)
[1:33] * oms101 (~oms101@p20030057EA00E700C6D987FFFE4339A1.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[1:33] <doppelgrau> https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/ <- might also benchmark your SSD directly
[1:33] <axion_joey> we should have gone that route. We went all ssd's thinking that it would give us better performance
[1:34] <doppelgrau> if theyre too slow with sync writes an option besides replacing might be one fast 8and durable)SSD for journal
[1:34] <axion_joey> each journal is on the same disk as storage
[1:34] * [0x4A6F]_ (~ident@p508CD483.dip0.t-ipconnect.de) has joined #ceph
[1:35] <doppelgrau> axion_joey: depending on the workload a erase coded platter storage with an (good) SSD-cache tier can be nice too
[1:35] * [0x4A6F] (~ident@0x4a6f.user.oftc.net) Quit (Ping timeout: 480 seconds)
[1:35] * [0x4A6F]_ is now known as [0x4A6F]
[1:38] <axion_joey> thanks so much for sharing the link. I'll benchmark our drives tonight
[1:39] <doppelgrau> and try to do a good read benchmark (either drop caches, or if both clusters are in use, just wait some time between write and read, so the data is moved outside the ram on the osd-servers
[1:39] <axion_joey> will do. They're not in production yet, so I can do that at anytime.
[1:40] <axion_joey> Is it possible to move the journals to different drives without wiping out and recreating the entire cluster?
[1:40] <doppelgrau> if the reads are still similarfast, something is really broken (even the cheapest SSDs are usually quite fast reading data)
[1:41] <doppelgrau> axion_joey: with some fideling you only have to take the osd down, (manual flusch journal, some ???black magic??? commands I dont remeber to add new journal, restart osd)
[1:41] <axion_joey> whoa
[1:41] * oms101 (~oms101@p20030057EA009900C6D987FFFE4339A1.dip0.t-ipconnect.de) has joined #ceph
[1:41] <axion_joey> I'll do a lot of googling :)
[1:42] <doppelgrau> axion_joey: or a bit more data movement but simpler: reweight to zer0, wait all data is moved. remove OSD from cluster, install new with journal, wait data to fill in, go to next OSD
[1:42] <axion_joey> that's so much better :)
[1:44] <axion_joey> doppelgrau, just so I'm 100% on this. I need to focus on the hardware/network/OS right? There are no ceph performance tuning variables?
[1:44] * georgem (~Adium@69-165-135-139.dsl.teksavvy.com) has joined #ceph
[1:46] <doppelgrau> axion_joey: there are some (in the archive of the mailinglist you might find some better hints, like number of concurent operations, IO thread ..), but on your level of performance, the reason should be somewhere else
[1:46] <axion_joey> got it
[1:47] <axion_joey> I know you can't give me a 100% answer, but what level of performance should I be shooting for?
[1:47] * linuxkidd (~linuxkidd@ip70-189-214-97.lv.lv.cox.net) Quit (Ping timeout: 480 seconds)
[1:47] * EinstCrazy (~EinstCraz@180.174.50.254) has joined #ceph
[1:47] <axion_joey> I'm at 165MB/s now. Should I shoot for 200, 300, 500?
[1:48] <doppelgrau> axion_joey: what is the ???size??? of your test cluster? (2 or three?)
[1:48] * EinstCrazy (~EinstCraz@180.174.50.254) Quit (Remote host closed the connection)
[1:48] <axion_joey> 8 hosts with 4 drives each
[1:49] <doppelgrau> I mean the number of replicas
[1:50] <doppelgrau> assuming three, I???d say you should be able to saturate your 2x10 Gbit ethernet links
[1:50] <axion_joey> sorry
[1:50] <axion_joey> 3 replicas
[1:51] <axion_joey> wow
[1:51] <doppelgrau> 2000 MB/s times three, divided by 32 OSDS times two writes
[1:52] <axion_joey> we are really far off
[1:52] <doppelgrau> 350MB/s, on each OSD, should be possible with 4MB objects rados bench uses
[1:52] <axion_joey> thanks for the info. I truly appreciate it
[1:52] <doppelgrau> (if you benchmark with 4k objects, the number of yourse drops way down)
[1:53] * jclm (~jclm@ip68-96-196-245.lv.lv.cox.net) Quit (Quit: Leaving.)
[1:53] <axion_joey> according to Sandisk the drives are rated for up to 515 MB/s
[1:53] <axion_joey> could the problem be that they're SATA and not SAS so having the journal on them is creating a huge bottleneck?
[1:54] <doppelgrau> axion_joey: the journal had been calculated into that numer (therefore the two writes, first journal, second storage)
[1:54] <axion_joey> got it
[1:55] <doppelgrau> either they do not ???like??? the synch IO from the journal => speed drops way down, or there is some other bottleneck
[1:56] <axion_joey> any chance you do consulting?
[1:56] * linuxkidd (~linuxkidd@107.72.96.44) has joined #ceph
[1:57] <doppelgrau> axion_joey: just to be sure: you do not have a ???private??? ceph network on a slower network setup, and only the publich on 10gbit?
[1:58] <axion_joey> that is correct. the ceph private network is on a 1gig network
[1:58] <doppelgrau> problem found
[1:58] <axion_joey> seriously?
[1:59] <axion_joey> i thought that network was just used to sync data when osd's went down or something
[1:59] * wushudoin (~wushudoin@2601:646:8200:c9f0:2ab2:bdff:fe0b:a6ee) Quit (Quit: Leaving)
[1:59] <doppelgrau> write goes over 10g to primary OSD, primary OSD copies data to the two other osds over private network ???
[1:59] * wushudoin (~wushudoin@2601:646:8200:c9f0:2ab2:bdff:fe0b:a6ee) has joined #ceph
[2:00] <axion_joey> wow
[2:00] <axion_joey> I'm an idiot
[2:00] <doppelgrau> and after the writes are acked on both other OSDs the primary OSDs acks the client...
[2:00] <axion_joey> send me a link to your amazon wish list
[2:00] <axion_joey> seriously
[2:00] <axion_joey> you saved me days of trial and error
[2:01] * wushudoin (~wushudoin@2601:646:8200:c9f0:2ab2:bdff:fe0b:a6ee) Quit ()
[2:01] * wushudoin (~wushudoin@2601:646:8200:c9f0:2ab2:bdff:fe0b:a6ee) has joined #ceph
[2:03] * wushudoin (~wushudoin@2601:646:8200:c9f0:2ab2:bdff:fe0b:a6ee) Quit ()
[2:03] <doppelgrau> no problem
[2:04] <doppelgrau> hope that fixes your problem. I???ll go to sleep now, getting (very) late here
[2:06] * salwasser (~Adium@2601:197:101:5cc1:41d0:c4ac:fd2b:745d) has joined #ceph
[2:06] * Concubidated (~cube@68.140.239.164) Quit (Quit: Leaving.)
[2:08] * andreww (~xarses@64.124.158.3) Quit (Ping timeout: 480 seconds)
[2:10] * georgem (~Adium@69-165-135-139.dsl.teksavvy.com) Quit (Quit: Leaving.)
[2:12] * doppelgrau (~doppelgra@dslb-088-072-094-200.088.072.pools.vodafone-ip.de) Quit (Quit: doppelgrau)
[2:12] * dicko (~Spessu@tor-exit.squirrel.theremailer.net) has joined #ceph
[2:15] * vicente (~vicente@1-161-184-59.dynamic.hinet.net) has joined #ceph
[2:22] * georgem (~Adium@69-165-135-139.dsl.teksavvy.com) has joined #ceph
[2:23] * vicente (~vicente@1-161-184-59.dynamic.hinet.net) Quit (Ping timeout: 480 seconds)
[2:24] * georgem (~Adium@69-165-135-139.dsl.teksavvy.com) Quit ()
[2:25] * georgem (~Adium@69-165-135-139.dsl.teksavvy.com) has joined #ceph
[2:28] * Skaag (~lunix@65.200.54.234) Quit (Quit: Leaving.)
[2:29] * andreww (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) has joined #ceph
[2:29] * georgem (~Adium@69-165-135-139.dsl.teksavvy.com) Quit ()
[2:34] * ircolle (~Adium@2601:285:201:633a:84fb:9944:b8d9:301e) has joined #ceph
[2:37] * davidzlap (~Adium@2605:e000:1313:8003:8d48:10c3:d607:89c8) Quit (Quit: Leaving.)
[2:38] * Concubidated (~cube@12.207.21.2) has joined #ceph
[2:38] * blizzow (~jburns@50-78-82-101-static.hfc.comcastbusiness.net) has joined #ceph
[2:40] * davidzlap (~Adium@2605:e000:1313:8003:cc43:ae71:8fc1:a16a) has joined #ceph
[2:42] * dicko (~Spessu@tor-exit.squirrel.theremailer.net) Quit ()
[2:48] * ira (~ira@c-24-34-255-34.hsd1.ma.comcast.net) Quit (Quit: Leaving)
[2:54] * blizzow (~jburns@50-78-82-101-static.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[3:01] * Green (~Green@221.178.90.226) has joined #ceph
[3:01] * jfaj (~jan@p4FC5B73A.dip0.t-ipconnect.de) has joined #ceph
[3:05] * vata (~vata@96.127.202.136) Quit (Quit: Leaving.)
[3:06] * rakeshgm (~rakesh@38.140.108.5) Quit (Quit: Peace :))
[3:08] * jfaj__ (~jan@p4FC5BB3D.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[3:09] * salwasser (~Adium@2601:197:101:5cc1:41d0:c4ac:fd2b:745d) Quit (Quit: Leaving.)
[3:13] * georgem (~Adium@69-165-135-139.dsl.teksavvy.com) has joined #ceph
[3:18] * Roy (~vend3r@46.166.138.144) has joined #ceph
[3:22] * vata (~vata@96.127.202.136) has joined #ceph
[3:34] * davidzlap (~Adium@2605:e000:1313:8003:cc43:ae71:8fc1:a16a) Quit (Quit: Leaving.)
[3:34] * xinli (~charleyst@32.97.110.56) has joined #ceph
[3:35] * EinstCrazy (~EinstCraz@58.247.119.250) has joined #ceph
[3:35] * Skaag (~lunix@cpe-172-91-77-84.socal.res.rr.com) has joined #ceph
[3:38] * haplo37 (~haplo37@199.91.185.156) Quit (Remote host closed the connection)
[3:48] * Roy (~vend3r@46.166.138.144) Quit ()
[3:52] * davidzlap (~Adium@cpe-172-91-154-245.socal.res.rr.com) has joined #ceph
[3:52] <Green> HA iSCSI
[3:54] * Bratwurtz (~Bratwurtz@103.42.225.249) has joined #ceph
[3:55] * vicente (~vicente@1-161-184-59.dynamic.hinet.net) has joined #ceph
[3:57] * wes_dillingham (~wes_dilli@209-6-222-74.c3-0.hdp-ubr1.sbo-hdp.ma.cable.rcn.com) has joined #ceph
[3:59] <Bratwurtz> Hello, we have an existing cluster with 3 mons, and we're looking to add a 4th. Does it matter if we add the monitor and then update the ceph.conf or do we have to update the ceph.conf first before we add the new monitor?
[3:59] <Bratwurtz> we were just going to use ceph-deploy mon add <host> to add the new monitor.
[4:00] * jfaj_ (~jan@p4FD26CF1.dip0.t-ipconnect.de) has joined #ceph
[4:00] <llua> Green: were you referring to this earlier? https://github.com/SUSE/lrbd/wiki
[4:05] * Skaag (~lunix@cpe-172-91-77-84.socal.res.rr.com) Quit (Quit: Leaving.)
[4:06] * jfaj (~jan@p4FC5B73A.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[4:23] * scg (~zscg@181.122.0.60) Quit (Quit: Ex-Chat)
[4:31] * scuttlemonkey is now known as scuttle|afk
[4:36] * tx (~oftc-webi@74.213.104.104) has joined #ceph
[4:37] <tx> How to remove ceph metada pools
[4:37] <tx> ?
[4:37] <tx> Error EBUSY: pool 'cephfs_data' is in use by CephFS
[4:40] * yanzheng (~zhyan@125.70.21.68) has joined #ceph
[4:42] * Skaag (~lunix@cpe-172-91-77-84.socal.res.rr.com) has joined #ceph
[4:51] <rkeene> tx, You have to delete the filesystem first
[4:53] <tx> root@mgmt:~# ceph mds fail 0 root@mgmt:~# ceph fs rm cephfs --yes-i-really-mean-it correct ?
[4:58] <rkeene> If you really mean it
[4:59] <tx> when i try of uninstall Error EBUSY: pool 'cephfs_data' is in use by CephFS
[5:01] * davidzlap (~Adium@cpe-172-91-154-245.socal.res.rr.com) Quit (Quit: Leaving.)
[5:02] * doppelgrau (~doppelgra@132.252.235.172) has joined #ceph
[5:03] * vicente (~vicente@1-161-184-59.dynamic.hinet.net) Quit (Ping timeout: 480 seconds)
[5:03] * wes_dillingham (~wes_dilli@209-6-222-74.c3-0.hdp-ubr1.sbo-hdp.ma.cable.rcn.com) Quit (Quit: wes_dillingham)
[5:06] * wes_dillingham (~wes_dilli@209-6-222-74.c3-0.hdp-ubr1.sbo-hdp.ma.cable.rcn.com) has joined #ceph
[5:07] * Vacuum__ (~Vacuum@88.130.207.127) has joined #ceph
[5:09] * georgem (~Adium@69-165-135-139.dsl.teksavvy.com) Quit (Quit: Leaving.)
[5:10] <m0zes> the cephfs daemon is stopped, right?
[5:12] <m0zes> s/cephfs/ceph-mds/
[5:14] * Vacuum_ (~Vacuum@88.130.196.44) Quit (Ping timeout: 480 seconds)
[5:14] <tx> althoug stop the cephfs daemon the pool can't be removed
[5:16] * linuxkidd (~linuxkidd@107.72.96.44) Quit (Ping timeout: 480 seconds)
[5:17] * dcwangmit01_ (~dcwangmit@162-245.23-239.PUBLIC.monkeybrains.net) has joined #ceph
[5:23] * dcwangmit01 (~dcwangmit@162-245.23-239.PUBLIC.monkeybrains.net) Quit (Ping timeout: 480 seconds)
[5:27] * davidzlap (~Adium@cpe-172-91-154-245.socal.res.rr.com) has joined #ceph
[5:27] * davidzlap (~Adium@cpe-172-91-154-245.socal.res.rr.com) Quit ()
[5:29] * wes_dillingham (~wes_dilli@209-6-222-74.c3-0.hdp-ubr1.sbo-hdp.ma.cable.rcn.com) Quit (Quit: wes_dillingham)
[5:33] * linuxkidd (~linuxkidd@ip70-189-214-97.lv.lv.cox.net) has joined #ceph
[5:33] * Green (~Green@221.178.90.226) Quit (Ping timeout: 480 seconds)
[5:35] <m0zes> key steps: 1) stop all cephfs daemons. 2) fail the "active" mds. 3) remove the cephfs filesystem. 4) then, you can remove the pools. I did this about a month ago. ceph mds dump will show you the status of the mdsmap as you move through the steps.
[5:36] <m0zes> if the daemon isn't stopped, failing mds 0 will cause it to restart and become active again. if the the mds isn't failed, you can't remove the cephfs filesystem, if the filesystem isn't removed, you can't remove data/metadata pools...
[5:42] * Green (~Green@221.178.90.226) has joined #ceph
[5:55] * vicente (~vicente@1-161-184-59.dynamic.hinet.net) has joined #ceph
[6:00] * vimal (~vikumar@114.143.162.209) has joined #ceph
[6:03] * vimal (~vikumar@114.143.162.209) Quit ()
[6:05] * vicente (~vicente@1-161-184-59.dynamic.hinet.net) Quit (Ping timeout: 480 seconds)
[6:09] * vata (~vata@96.127.202.136) Quit (Quit: Leaving.)
[6:11] * walcubi (~walcubi@p5795A3CD.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[6:11] * walcubi (~walcubi@p5797A30D.dip0.t-ipconnect.de) has joined #ceph
[6:17] * toastyde2th (~toast@pool-71-255-253-39.washdc.fios.verizon.net) Quit (Read error: Connection reset by peer)
[6:17] * toastyde2th (~toast@pool-71-255-253-39.washdc.fios.verizon.net) has joined #ceph
[6:19] * jarrpa (~jarrpa@167.220.99.192) Quit (Ping timeout: 480 seconds)
[6:26] * vimal (~vikumar@121.244.87.116) has joined #ceph
[6:29] * Green (~Green@221.178.90.226) Quit (Ping timeout: 480 seconds)
[6:35] * ircolle (~Adium@2601:285:201:633a:84fb:9944:b8d9:301e) Quit (Quit: Leaving.)
[6:38] * ivve (~zed@cust-gw-11.se.zetup.net) has joined #ceph
[6:41] * Hemanth (~hkumar_@103.228.221.140) has joined #ceph
[6:56] * tx (~oftc-webi@74.213.104.104) Quit (Remote host closed the connection)
[6:58] * vicente (~vicente@1-161-184-59.dynamic.hinet.net) has joined #ceph
[7:00] * vimal (~vikumar@121.244.87.116) Quit (Quit: Leaving)
[7:04] * vimal (~vikumar@121.244.87.116) has joined #ceph
[7:04] * mnc (~mnc@c-50-137-214-131.hsd1.mn.comcast.net) has joined #ceph
[7:06] * vicente (~vicente@1-161-184-59.dynamic.hinet.net) Quit (Ping timeout: 480 seconds)
[7:08] * rdas (~rdas@121.244.87.116) has joined #ceph
[7:19] * sudocat (~dibarra@2602:306:8bc7:4c50:46a:618e:384b:72a) has joined #ceph
[7:21] * wjw-freebsd (~wjw@smtp.digiware.nl) Quit (Ping timeout: 480 seconds)
[7:28] * jarrpa (~jarrpa@216-9-6-126.hfn.isomedia.com) has joined #ceph
[7:42] * flisky (~Thunderbi@210.12.157.85) has joined #ceph
[7:47] * Concubidated1 (~cube@66.87.118.60) has joined #ceph
[7:54] * Concubidated (~cube@12.207.21.2) Quit (Ping timeout: 480 seconds)
[7:56] * Schaap (~vend3r@108.61.122.221) has joined #ceph
[7:59] * vicente (~vicente@1-161-184-59.dynamic.hinet.net) has joined #ceph
[7:59] * jarrpa (~jarrpa@216-9-6-126.hfn.isomedia.com) Quit (Remote host closed the connection)
[8:07] * vicente (~vicente@1-161-184-59.dynamic.hinet.net) Quit (Ping timeout: 480 seconds)
[8:24] * rraja (~rraja@125.16.34.66) has joined #ceph
[8:24] * Be-El (~blinke@nat-router.computational.bio.uni-giessen.de) has joined #ceph
[8:26] * Schaap (~vend3r@108.61.122.221) Quit ()
[8:26] * sardonyx (~Guest1390@178.175.128.50) has joined #ceph
[8:30] * Skaag (~lunix@cpe-172-91-77-84.socal.res.rr.com) Quit (Quit: Leaving.)
[8:33] * derjohn_mob (~aj@p578b6aa1.dip0.t-ipconnect.de) has joined #ceph
[8:34] * karnan (~karnan@125.16.34.66) has joined #ceph
[8:40] * verdurin (~verdurin@2001:8b0:281:78ec:e2cb:4eff:fe01:f767) has joined #ceph
[8:49] * sudocat (~dibarra@2602:306:8bc7:4c50:46a:618e:384b:72a) Quit (Ping timeout: 480 seconds)
[8:50] * xinli (~charleyst@32.97.110.56) Quit (Ping timeout: 480 seconds)
[8:56] * dgurtner (~dgurtner@84-73-130-19.dclient.hispeed.ch) has joined #ceph
[8:56] * sardonyx (~Guest1390@178.175.128.50) Quit ()
[8:57] * ade (~abradshaw@2a02:810d:a4c0:5cd:9001:2bba:886a:5200) has joined #ceph
[8:58] * analbeard (~shw@support.memset.com) has joined #ceph
[8:59] * vicente (~vicente@1-161-184-59.dynamic.hinet.net) has joined #ceph
[9:01] * doppelgrau (~doppelgra@132.252.235.172) Quit (Quit: Leaving.)
[9:03] * blizzow (~jburns@c-50-152-51-96.hsd1.co.comcast.net) has joined #ceph
[9:07] * fridim (~fridim@56-198-190-109.dsl.ovh.fr) has joined #ceph
[9:07] * vicente (~vicente@1-161-184-59.dynamic.hinet.net) Quit (Ping timeout: 480 seconds)
[9:09] * flisky1 (~Thunderbi@106.38.61.185) has joined #ceph
[9:11] * flisky (~Thunderbi@210.12.157.85) Quit (Remote host closed the connection)
[9:11] * flisky1 is now known as flisky
[9:13] * T1w (~jens@node3.survey-it.dk) has joined #ceph
[9:17] * blizzow (~jburns@c-50-152-51-96.hsd1.co.comcast.net) Quit (Ping timeout: 480 seconds)
[9:23] * wjw-freebsd (~wjw@smtp.digiware.nl) has joined #ceph
[9:23] * Concubidated1 (~cube@66.87.118.60) Quit (Quit: Leaving.)
[9:23] * dgurtner (~dgurtner@84-73-130-19.dclient.hispeed.ch) Quit (Quit: leaving)
[9:23] * dgurtner (~dgurtner@84-73-130-19.dclient.hispeed.ch) has joined #ceph
[9:27] * Hemanth (~hkumar_@103.228.221.140) Quit (Ping timeout: 480 seconds)
[9:28] * Hemanth (~hkumar_@103.228.221.140) has joined #ceph
[9:28] * fridim (~fridim@56-198-190-109.dsl.ovh.fr) Quit (Ping timeout: 480 seconds)
[9:33] * flisky (~Thunderbi@106.38.61.185) Quit (Quit: flisky)
[9:34] * owlbot (~supybot@pct-empresas-50.uc3m.es) Quit (Remote host closed the connection)
[9:35] * owlbot (~supybot@pct-empresas-50.uc3m.es) has joined #ceph
[9:35] * Lokta (~Lokta@carbon.coe.int) has joined #ceph
[9:39] * fridim (~fridim@56-198-190-109.dsl.ovh.fr) has joined #ceph
[9:45] * kefu (~kefu@114.92.125.128) has joined #ceph
[9:49] <peetaur2_> Bratwurtz: I think the order is not so strict
[9:50] <peetaur2_> it possibly doesn't matter until the mon is up and talks to the rest of the cluster
[9:50] <peetaur2_> but I don't have any proof of that
[9:53] * raphaelsc (~raphaelsc@177.134.123.77) Quit (Remote host closed the connection)
[10:00] * mnc (~mnc@c-50-137-214-131.hsd1.mn.comcast.net) Quit (Quit: Leaving)
[10:00] * nils_ (~nils_@doomstreet.collins.kg) has joined #ceph
[10:01] * DanFoster (~Daniel@2a00:1ee0:3:1337:1000:2cd1:f412:5d37) has joined #ceph
[10:01] * doppelgrau (~doppelgra@dslb-088-072-094-200.088.072.pools.vodafone-ip.de) has joined #ceph
[10:18] * karnan_ (~karnan@125.16.34.66) has joined #ceph
[10:19] * derjohn_mob (~aj@p578b6aa1.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[10:25] * karnan (~karnan@125.16.34.66) Quit (Ping timeout: 480 seconds)
[10:27] * TMM (~hp@dhcp-077-248-009-229.chello.nl) Quit (Quit: Ex-Chat)
[10:30] * vicente (~vicente@1-161-184-59.dynamic.hinet.net) has joined #ceph
[10:30] * kefu (~kefu@114.92.125.128) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[10:38] * vicente (~vicente@1-161-184-59.dynamic.hinet.net) Quit (Ping timeout: 480 seconds)
[10:40] * kefu (~kefu@114.92.125.128) has joined #ceph
[10:48] * Jaska (~Jaska@64.137.246.130) has joined #ceph
[10:49] <Bratwurtz> peetaur2_: So the monitor can be added into the cluster and form a quorum, but the OSDs/Clients will simply ignore it until their mon_hosts values are updated to use it?
[10:49] <Bratwurtz> Kind of makes sense :)
[10:49] <peetaur2_> it's possible that the client will try to use that monitor before it's up, and then delay and then revert to another
[10:49] * derjohn_mob (~aj@46.189.28.56) has joined #ceph
[10:49] * Be-El (~blinke@nat-router.computational.bio.uni-giessen.de) has left #ceph
[10:50] * Be-El (~blinke@nat-router.computational.bio.uni-giessen.de) has joined #ceph
[10:50] <peetaur2_> but the quorum doesn't seem to be affected until you add the node (ceph mon add "$id" "$ip", which ceph-deploy probably does) and maybe also the node has started
[10:52] <Bratwurtz> Thanks. So if we add the monitor and then update the configs we should be ok?
[10:52] * ivve (~zed@cust-gw-11.se.zetup.net) Quit (Ping timeout: 480 seconds)
[10:55] * ivve (~zed@cust-gw-11.se.zetup.net) has joined #ceph
[10:56] * fdmanana (~fdmanana@2001:8a0:6e0c:6601:1cec:4075:a3a:1bce) has joined #ceph
[10:57] <peetaur2_> do you use ceph-deploy?
[10:57] <peetaur2_> what I did in my testing was all manual, no ceph deploy, and I started the mon first, then added to mon map, then added to the new mon's ceph.conf, then copied the new ceph.conf to other nodes
[10:58] <peetaur2_> which is here http://docs.ceph.com/docs/jewel/install/manual-deployment/#monitor-bootstrapping
[10:59] <Bratwurtz> Yes, we use ceph-deploy, which i think (if we use mon add) takes care of starting the monitor and adding it to mon map. So we should be good to just update the conf's and off we go... I really should get a lab.
[10:59] <Bratwurtz> Thanks for the info.
[11:04] * wjw-freebsd (~wjw@smtp.digiware.nl) Quit (Read error: Connection reset by peer)
[11:04] * wjw-freebsd (~wjw@smtp.digiware.nl) has joined #ceph
[11:07] <peetaur2_> making a virtual lab is easy... do it :P
[11:14] * rdas (~rdas@121.244.87.116) Quit (Quit: Leaving)
[11:18] * Jaska (~Jaska@64.137.246.130) Quit ()
[11:20] * TMM (~hp@185.5.121.201) has joined #ceph
[11:21] * TomasCZ (~TomasCZ@yes.tenlab.net) Quit (Ping timeout: 480 seconds)
[11:22] * kefu (~kefu@114.92.125.128) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[11:25] * TomasCZ (~TomasCZ@yes.tenlab.net) has joined #ceph
[11:28] * rdas (~rdas@121.244.87.113) has joined #ceph
[11:35] * gong (~oftc-webi@89-160-187-133.du.xdsl.is) has joined #ceph
[11:36] <gong> Hi guys, I'm having some issues with my pgs
[11:37] <gong> http://pastebin.com/01XgAQ4w
[11:37] <gong> Any steps I can take from here?
[11:40] * kefu (~kefu@114.92.125.128) has joined #ceph
[11:40] <peetaur2_> how about ceph -s without the line endings removed
[11:43] <gong> peetaur2_: just changed my pool sizes to 1, to see if that helps things
[11:43] <gong> but here's ceph -s http://pastebin.com/1vShtqLA
[11:44] <peetaur2_> ok so the "70 pgs down" is gone now, and I think it looks like something is supposed to happen... do the % numbers change?
[11:44] <peetaur2_> also there's ceph -w to watch it
[11:45] * doppelgrau (~doppelgra@dslb-088-072-094-200.088.072.pools.vodafone-ip.de) Quit (Quit: doppelgrau)
[11:47] <gong> the % doesn't seem to change
[11:47] <gong> but there seems to be some scrubbing going on right now
[11:47] <gong> and a lot of slow request messages
[11:48] <peetaur2_> have you read http://docs.ceph.com/docs/jewel/rados/troubleshooting/troubleshooting-pg/ ?
[11:49] <gong> peetaur2_: yep I read that last night, added a new osd, etc. Left it for the night only to see the ceph health exactly the same in the morning
[11:50] <arthur> hi, is it possible to move a journal from ssd back to a spinning hdd? Or would I have to remove and re-create the osd managing the hdd?
[11:50] <peetaur2_> arthur: I did that in my tests :)
[11:51] <arthur> since the hdd has no separate partition for journal I assume I have to recreate it
[11:51] <peetaur2_> arthur: just stop osd, dd from old to new location, update ceph.conf, start osd
[11:51] <peetaur2_> don't know if that works to go from block to file or back though
[11:51] <peetaur2_> I went from block to block
[11:52] <peetaur2_> if it's just a test cluster, try it for fun :)
[11:52] <arthur> ah, so then /var/lib/ceph/osd/ceph-0/journal becomes a file instead of a symlink
[11:52] <peetaur2_> and you'd think there's a way to recover an osd without the journal, which would also do that, but I don't know how
[11:52] <peetaur2_> is it a symlink for you?
[11:53] <arthur> yeah, to /dev/disk/by-partuuid
[11:53] <peetaur2_> I did a manual deployment and the docs didn't say what to do, so I made a symlink, which worked great in hammer, but then in jewel it fails due to running as ceph instead of root, so I put the location in ceph.conf instead, which worked
[11:53] <arthur> which is also a symlink to /dev/sdg1
[11:53] <arthur> I deployed with proxmox' pveceph utility
[11:55] <Be-El> arthur: you can also flush the journal after stopping the osd and simply change the symlink and create a new journal
[11:56] * derjohn_mob (~aj@46.189.28.56) Quit (Ping timeout: 480 seconds)
[11:56] <Be-El> arthur: with that procedure you do not need to copy the data between devices
[11:57] <peetaur2_> ah, so ceph-osd --flush-journal ...
[11:57] <peetaur2_> I suspected such a thing would exist
[11:57] <arthur> Be-El: I figured the same, I did the same thing when replacing my former journal ssd
[11:57] <arthur> thanks :)
[11:59] * dugravot6 (~dugravot6@l-p-dn-in-4a.lionnois.site.univ-lorraine.fr) Quit (Quit: Leaving.)
[12:01] * vicente (~vicente@1-161-184-59.dynamic.hinet.net) has joined #ceph
[12:03] <gong> peetaur2_: I noticed in my crushmap all the devices are called osd.0, osd.1, etc. But one device is called device4. That can't be normal, can it?
[12:04] * derjohn_mob (~aj@46.189.28.35) has joined #ceph
[12:04] * dugravot6 (~dugravot6@l-p-dn-in-4a.lionnois.site.univ-lorraine.fr) has joined #ceph
[12:04] <peetaur2_> gong: heh sounds very wrong. pastebin the map?
[12:06] <peetaur2_> I remember trying to name an osd after the host, like node1-0 and it didn't like it... it wanted a number and said it might work but is deprecated or some word like that
[12:06] <gong> peetaur2_: here's the crush map http://pastebin.com/f0BMkgzE
[12:06] <gong> and here are the osd's http://pastebin.com/vtq2xnD7
[12:08] * EinstCrazy (~EinstCraz@58.247.119.250) Quit (Remote host closed the connection)
[12:08] * EinstCrazy (~EinstCraz@58.247.119.250) has joined #ceph
[12:09] <peetaur2_> gong: is there a pve05?
[12:09] <gong> peetaur2_: pve05 only has a monitor, no osd's
[12:09] * vicente (~vicente@1-161-184-59.dynamic.hinet.net) Quit (Ping timeout: 480 seconds)
[12:09] <peetaur2_> ok so seems device4 is not in any host, so I guess it's not a real osd
[12:10] <gong> so it should be safe to remove?
[12:10] <peetaur2_> not sure... maybe wait until the other problem is clear
[12:11] * kefu (~kefu@114.92.125.128) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[12:11] <peetaur2_> and I forgot to ask before... did anything fail before this happened? like an osd died and you replaced it, or you played around with size?
[12:13] <gong> Well what happened was that I was rebooting a node, it took longer time than expected so the recovery seems to have kicked off
[12:13] <gong> and that filled the osd's one by one
[12:13] <gong> 100% full disk space, so the osd's got set to stopped/out
[12:14] <gong> When the node was back up I took a look at the 100% full osd's and removed some pg's that I assumed was safe to remove
[12:14] <gong> To get the osd's back up
[12:15] <gong> got all of them back up except for one that I had to zap and re-create
[12:16] * EinstCrazy (~EinstCraz@58.247.119.250) Quit (Ping timeout: 480 seconds)
[12:26] * doppelgrau (~doppelgra@132.252.235.172) has joined #ceph
[12:32] <peetaur2_> ok so sounds like a mess :)
[12:33] <peetaur2_> does ceph osd df show that this one new osd has little data?
[12:34] * baotiao (~baotiao@218.30.116.7) has joined #ceph
[12:37] <gong> peetaur2_: yep very little data
[12:38] <gong> and the IO wait is high, can't get my rbd vm's running
[12:38] <gong> can't unmap rbd volume vm-109-disk-1: rbd: sysfs write failed
[12:45] * analbeard (~shw@support.memset.com) Quit (Quit: Leaving.)
[12:49] * wjw-freebsd (~wjw@smtp.digiware.nl) Quit (Ping timeout: 480 seconds)
[12:52] <peetaur2_> and you're sure it was only one osd that died?
[12:53] <TMM> I appear to have objects on my OSDs for rbd images I can't find
[12:53] <TMM> is that normal?
[12:53] <TMM> I have rbd_data.66c7d522a3d33c.... files
[12:54] <TMM> but I can't find any reference to them when I do an rbd info on all my images
[12:56] * rmart04 (~rmart04@support.memset.com) has joined #ceph
[13:09] * rdas (~rdas@121.244.87.113) Quit (Ping timeout: 480 seconds)
[13:13] * analbeard (~shw@support.memset.com) has joined #ceph
[13:14] * georgem (~Adium@24.114.55.233) has joined #ceph
[13:18] * toastyde2th (~toast@pool-71-255-253-39.washdc.fios.verizon.net) Quit (Read error: Connection reset by peer)
[13:19] * georgem (~Adium@24.114.55.233) Quit (Quit: Leaving.)
[13:19] * georgem (~Adium@206.108.127.16) has joined #ceph
[13:21] * dcwangmit01_ (~dcwangmit@162-245.23-239.PUBLIC.monkeybrains.net) Quit (Remote host closed the connection)
[13:23] * dcwangmit01 (~dcwangmit@162-245.23-239.PUBLIC.monkeybrains.net) has joined #ceph
[13:24] * bniver (~bniver@pool-98-110-180-234.bstnma.fios.verizon.net) Quit (Remote host closed the connection)
[13:26] * kuku (~kuku@112.203.19.176) has joined #ceph
[13:45] * Racpatel (~Racpatel@2601:87:3:31e3::4d2a) has joined #ceph
[13:47] * vicente (~vicente@1-161-184-59.dynamic.hinet.net) has joined #ceph
[13:47] * kuku (~kuku@112.203.19.176) Quit (Remote host closed the connection)
[13:47] * ade (~abradshaw@2a02:810d:a4c0:5cd:9001:2bba:886a:5200) Quit (Remote host closed the connection)
[13:51] * ade (~abradshaw@2a02:810d:a4c0:5cd:9001:2bba:886a:5200) has joined #ceph
[13:52] * dgurtner (~dgurtner@84-73-130-19.dclient.hispeed.ch) Quit (Quit: leaving)
[13:52] * dgurtner (~dgurtner@84-73-130-19.dclient.hispeed.ch) has joined #ceph
[13:54] * JANorman (~JANorman@81.137.246.31) has joined #ceph
[13:55] * vicente (~vicente@1-161-184-59.dynamic.hinet.net) Quit (Ping timeout: 480 seconds)
[13:56] <JANorman> Hi guys, when I try to do a rados_append to an object in an erasure coded pool, I get error 95, operation not permitted. I've read this is to do with stripe_width alignment, and I know there are API functions to find out if alignment is required, but not to sure on what to do if alignment is required? Would it be a configuration change in the ceph cluster?
[13:57] <JANorman> *operation not supported, sorry
[13:57] <TMM> I'm like, 99% sure that you can only do those operations on distributed pools, you may just want to add a small distributed pool as a cache tier in front of your ec pool
[13:58] <TMM> that way all operations should 'just work'
[13:59] * valeech (~valeech@pool-96-247-203-33.clppva.fios.verizon.net) has joined #ceph
[14:00] <JANorman> By distributed, do you mean replicated?
[14:00] <TMM> JANorman, yeah, sorry... I don't know where that came from
[14:01] <JANorman> TMM, And that way the replicated pool handles writing to the EC pool?
[14:02] * kefu (~kefu@114.92.125.128) has joined #ceph
[14:02] <TMM> JANorman, objects get moved back and forth between the ec and cache tiers based on demand
[14:02] <TMM> JANorman, it's transparent to the clients, for the most part you'll see reads of old(er) data going straight from the ec pools whereas writes will always go to the cache tiers and they get demoted to the background tier after a configurable time
[14:02] <TMM> at which point they will be written as ec objects
[14:03] <JANorman> I'm not an expert at all, but I'm guess that writing to the replicated pool is faster than writing to the ec pool?
[14:04] <TMM> that too
[14:04] <TMM> generally
[14:05] * Racpatel (~Racpatel@2601:87:3:31e3::4d2a) Quit (Ping timeout: 480 seconds)
[14:05] <JANorman> Gotcha - thanks a million
[14:05] <JANorman> :)
[14:06] <TMM> I've actually found that reading from ec pools on slower media can be faster sometimes than replicated pools
[14:09] * bniver (~bniver@71-9-144-29.static.oxfr.ma.charter.com) has joined #ceph
[14:09] * owasserm (~owasserm@2001:984:d3f7:1:5ec5:d4ff:fee0:f6dc) Quit (Remote host closed the connection)
[14:10] * wes_dillingham (~wes_dilli@140.247.242.44) has joined #ceph
[14:10] * realitysandwich (~perry@2001:920:1846:1dc0:baae:edff:fe73:f413) has joined #ceph
[14:11] * owasserm (~owasserm@2001:984:d3f7:1:5ec5:d4ff:fee0:f6dc) has joined #ceph
[14:11] <realitysandwich> I am having an issue right now where when one of my OSDs dies the machine keeps trying to send operations to it and they backlog and clog the machine until it crashes, is there some way to prevent this behavior?
[14:12] * owasserm_ (~owasserm@2001:984:d3f7:1:5ec5:d4ff:fee0:f6dc) has joined #ceph
[14:12] * owasserm__ (~owasserm@2001:984:d3f7:1:5ec5:d4ff:fee0:f6dc) has joined #ceph
[14:12] * georgem (~Adium@206.108.127.16) Quit (Ping timeout: 480 seconds)
[14:15] * ggarg (~ggarg@host-82-135-29-34.customer.m-online.net) Quit (Ping timeout: 480 seconds)
[14:15] <TMM> realitysandwich, does the osd get marked 'down'?
[14:19] <realitysandwich> no, it isnt being marked as down
[14:19] * madkiss2 (~madkiss@2a02:8109:8680:2000:d064:50ef:5247:4bdf) Quit (Quit: Leaving.)
[14:23] * georgem (~Adium@206.108.127.16) has joined #ceph
[14:25] * owasserm (~owasserm@2001:984:d3f7:1:5ec5:d4ff:fee0:f6dc) Quit (Remote host closed the connection)
[14:25] * owasserm_ (~owasserm@2001:984:d3f7:1:5ec5:d4ff:fee0:f6dc) Quit (Remote host closed the connection)
[14:25] * owasserm__ (~owasserm@2001:984:d3f7:1:5ec5:d4ff:fee0:f6dc) Quit (Remote host closed the connection)
[14:25] * owasserm (~owasserm@2001:984:d3f7:1:5ec5:d4ff:fee0:f6dc) has joined #ceph
[14:26] * kefu (~kefu@114.92.125.128) Quit (Max SendQ exceeded)
[14:27] * kefu (~kefu@li1456-173.members.linode.com) has joined #ceph
[14:30] * ira (~ira@c-24-34-255-34.hsd1.ma.comcast.net) has joined #ceph
[14:30] * owasserm (~owasserm@2001:984:d3f7:1:5ec5:d4ff:fee0:f6dc) Quit (Remote host closed the connection)
[14:33] * owasserm (~owasserm@2001:984:d3f7:1:5ec5:d4ff:fee0:f6dc) has joined #ceph
[14:34] <TMM> realitysandwich, you may want to ensure that your min down reports are all set correctly
[14:34] <TMM> realitysandwich, if osds aren't being marked down that's a problem
[14:35] * mattbenjamin1 (~mbenjamin@76-206-42-50.lightspeed.livnmi.sbcglobal.net) has joined #ceph
[14:35] * owasserm (~owasserm@2001:984:d3f7:1:5ec5:d4ff:fee0:f6dc) Quit (Remote host closed the connection)
[14:36] * owasserm (~owasserm@2001:984:d3f7:1:5ec5:d4ff:fee0:f6dc) has joined #ceph
[14:36] * mattbenjamin1 (~mbenjamin@76-206-42-50.lightspeed.livnmi.sbcglobal.net) Quit ()
[14:36] * pdrakewe_ (~pdrakeweb@oh-76-5-101-140.dhcp.embarqhsd.net) has joined #ceph
[14:38] * karnan_ (~karnan@125.16.34.66) Quit (Ping timeout: 480 seconds)
[14:41] * bauruine (~bauruine@2a01:4f8:130:8285:fefe::36) Quit (Quit: ZNC - http://znc.in)
[14:42] * bauruine (~bauruine@2a01:4f8:130:8285:fefe::36) has joined #ceph
[14:43] * pdrakeweb (~pdrakeweb@cpe-71-74-153-111.neo.res.rr.com) Quit (Ping timeout: 480 seconds)
[14:44] * bauruine (~bauruine@2a01:4f8:130:8285:fefe::36) Quit (Read error: Connection reset by peer)
[14:44] * bauruine (~bauruine@mail.tuxli.ch) has joined #ceph
[14:45] * mgolub (~mgolub@93.183.239.2) Quit (Ping timeout: 480 seconds)
[14:45] * rwheeler (~rwheeler@pool-108-7-196-31.bstnma.fios.verizon.net) has joined #ceph
[14:46] * ggarg (~ggarg@host-82-135-29-34.customer.m-online.net) has joined #ceph
[14:47] * vicente (~vicente@1-161-184-59.dynamic.hinet.net) has joined #ceph
[14:48] * karnan_ (~karnan@125.16.34.66) has joined #ceph
[14:55] * vicente (~vicente@1-161-184-59.dynamic.hinet.net) Quit (Ping timeout: 480 seconds)
[15:00] * mattbenjamin1 (~mbenjamin@76-206-42-50.lightspeed.livnmi.sbcglobal.net) has joined #ceph
[15:00] * baotiao (~baotiao@218.30.116.7) Quit (Quit: baotiao)
[15:04] * JANorman (~JANorman@81.137.246.31) Quit (Remote host closed the connection)
[15:04] * JANorman (~JANorman@81.137.246.31) has joined #ceph
[15:06] * JANorman (~JANorman@81.137.246.31) Quit ()
[15:11] * karnan_ (~karnan@125.16.34.66) Quit (Remote host closed the connection)
[15:12] * trociny (~mgolub@93.183.239.2) has joined #ceph
[15:16] * Bratwurtz (~Bratwurtz@103.42.225.249) Quit (Quit: Leaving...)
[15:21] * mattbenjamin1 (~mbenjamin@76-206-42-50.lightspeed.livnmi.sbcglobal.net) Quit (Ping timeout: 480 seconds)
[15:21] * salwasser (~Adium@72.246.3.14) has joined #ceph
[15:28] * xinli (~charleyst@32.97.110.54) has joined #ceph
[15:29] * John341 (~ceph@118.200.221.105) has joined #ceph
[15:30] * Racpatel (~Racpatel@2601:87:3:31e3::4d2a) has joined #ceph
[15:31] <John341> Hi, I'm CentOS7/Hammer and I have radosgw multipart and shadow objects in .rgw.buckets even though I have deleted all buckets 2weeks ago, can anybody advice on how to prune or garbage collect the orphan objects?
[15:31] * kefu (~kefu@li1456-173.members.linode.com) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[15:32] * vata (~vata@207.96.182.162) has joined #ceph
[15:32] * kefu (~kefu@li1456-173.members.linode.com) has joined #ceph
[15:32] * mattbenjamin1 (~mbenjamin@12.118.3.106) has joined #ceph
[15:35] * vicente (~vicente@1-161-184-59.dynamic.hinet.net) has joined #ceph
[15:45] <arthur> i want to delete 1 osd from each of my 4 ceph nodes: i first stop osd.x, then set osd.x out and wait for all pg to become active+clean
[15:45] <arthur> but one of the osd's is stuck at: 1 active+undersized+degraded
[15:48] * T1w (~jens@node3.survey-it.dk) Quit (Ping timeout: 480 seconds)
[15:48] * arsenaali (~Ian2128@108.61.122.216) has joined #ceph
[15:49] <TMM> arthur, you may want to check the min_size on the pool that that pg is a part of
[15:55] <peetaur2_> arthur: safer and faster is to set it out, then when it's clean, stop it
[15:56] <peetaur2_> arthur: out means you plan to clear it off..so you can clear it without being degraded also
[15:56] <John341> anyone who knows how the rgw gc runs, which process and how it is triggered?
[16:00] * Nicho1as (~nicho1as@00022427.user.oftc.net) has joined #ceph
[16:03] * Green (~Green@27.11.129.173) has joined #ceph
[16:12] * andreww (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[16:17] * briner (~briner@129.194.16.54) has joined #ceph
[16:18] * arsenaali (~Ian2128@108.61.122.216) Quit ()
[16:25] * andreww (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) has joined #ceph
[16:28] * ghosh (~ghosh@pd95b4e37.dip0.t-ipconnect.de) has joined #ceph
[16:29] * kefu_ (~kefu@114.92.125.128) has joined #ceph
[16:31] * ghosh (~ghosh@pd95b4e37.dip0.t-ipconnect.de) Quit ()
[16:33] <IcePic> isnt there some "run this every X seconds" which defaults to 3600
[16:34] * sudocat (~dibarra@104-188-116-197.lightspeed.hstntx.sbcglobal.net) has joined #ceph
[16:37] * kefu (~kefu@li1456-173.members.linode.com) Quit (Ping timeout: 480 seconds)
[16:38] * AlexeyAbashkin (~AlexeyAba@91.207.132.67) has joined #ceph
[16:39] * dgurtner (~dgurtner@84-73-130-19.dclient.hispeed.ch) Quit (Ping timeout: 480 seconds)
[16:42] * dgurtner (~dgurtner@84-73-130-19.dclient.hispeed.ch) has joined #ceph
[16:42] * sudocat (~dibarra@104-188-116-197.lightspeed.hstntx.sbcglobal.net) Quit (Ping timeout: 480 seconds)
[16:42] * jlayton (~jlayton@cpe-2606-A000-1125-405B-14D9-DFF4-8FF1-7DD8.dyn6.twc.com) Quit (Quit: ZNC 1.6.2 - http://znc.in)
[16:45] * jlayton (~jlayton@cpe-2606-A000-1125-405B-14D9-DFF4-8FF1-7DD8.dyn6.twc.com) has joined #ceph
[16:47] * haplo37 (~haplo37@199.91.185.156) has joined #ceph
[16:54] * sudocat (~dibarra@2602:306:8bc7:4c50:46a:618e:384b:72a) has joined #ceph
[16:57] * vikhyat (~vumrao@123.252.148.203) has joined #ceph
[16:57] * ntpttr_ (~ntpttr@134.134.139.77) has joined #ceph
[16:58] * rmart04 (~rmart04@support.memset.com) Quit (Quit: rmart04)
[17:00] * b0e (~aledermue@213.95.25.82) has joined #ceph
[17:00] * analbeard (~shw@support.memset.com) Quit (Quit: Leaving.)
[17:00] * georgem (~Adium@206.108.127.16) Quit (Read error: Connection reset by peer)
[17:00] * georgem (~Adium@206.108.127.16) has joined #ceph
[17:02] * Skaag (~lunix@cpe-172-91-77-84.socal.res.rr.com) has joined #ceph
[17:03] <John341> IcePic: is there a way to manually trigger the cleanups, and where does it get logged? :s
[17:04] * kefu_ is now known as kefu|afk
[17:05] * b0e (~aledermue@213.95.25.82) Quit (Quit: Leaving.)
[17:08] * yanzheng (~zhyan@125.70.21.68) Quit (Quit: This computer has gone to sleep)
[17:10] * Skaag (~lunix@cpe-172-91-77-84.socal.res.rr.com) Quit (Quit: Leaving.)
[17:11] * kefu|afk (~kefu@114.92.125.128) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[17:11] * georgem (~Adium@206.108.127.16) Quit (Ping timeout: 480 seconds)
[17:12] * rraja (~rraja@125.16.34.66) Quit (Ping timeout: 480 seconds)
[17:16] * Concubidated (~cube@68.140.239.164) has joined #ceph
[17:17] * ade (~abradshaw@2a02:810d:a4c0:5cd:9001:2bba:886a:5200) Quit (Ping timeout: 480 seconds)
[17:17] * wushudoin (~wushudoin@38.140.108.2) has joined #ceph
[17:19] * baotiao (~baotiao@43.255.178.177) has joined #ceph
[17:22] * sudocat (~dibarra@2602:306:8bc7:4c50:46a:618e:384b:72a) Quit (Remote host closed the connection)
[17:29] * TMM (~hp@185.5.121.201) Quit (Quit: Ex-Chat)
[17:30] * doppelgrau (~doppelgra@132.252.235.172) Quit (Quit: Leaving.)
[17:31] * Nicho1as (~nicho1as@00022427.user.oftc.net) Quit (Quit: A man from the Far East; using WeeChat 1.5)
[17:36] * Green (~Green@27.11.129.173) Quit (Ping timeout: 480 seconds)
[17:36] * ivve (~zed@cust-gw-11.se.zetup.net) Quit (Ping timeout: 480 seconds)
[17:38] * vicente (~vicente@1-161-184-59.dynamic.hinet.net) Quit (Ping timeout: 480 seconds)
[17:43] * Skaag (~lunix@cpe-172-91-77-84.socal.res.rr.com) has joined #ceph
[17:47] * derjohn_mob (~aj@46.189.28.35) Quit (Ping timeout: 480 seconds)
[17:48] * blank (~Joppe4899@exit0.radia.tor-relays.net) has joined #ceph
[17:50] * newbie (~kvirc@host217-114-156-249.pppoe.mark-itt.net) has joined #ceph
[17:54] * Lokta (~Lokta@carbon.coe.int) Quit (Ping timeout: 480 seconds)
[17:56] * vikhyat (~vumrao@123.252.148.203) Quit (Quit: Leaving)
[17:58] * scg (~zscg@181.122.0.60) has joined #ceph
[18:00] * Skaag (~lunix@cpe-172-91-77-84.socal.res.rr.com) Quit (Quit: Leaving.)
[18:02] * sudocat (~dibarra@192.185.1.20) has joined #ceph
[18:03] * vimal (~vikumar@121.244.87.116) Quit (Quit: Leaving)
[18:07] * doppelgrau (~doppelgra@dslb-088-072-094-200.088.072.pools.vodafone-ip.de) has joined #ceph
[18:18] * georgem (~Adium@206.108.127.16) has joined #ceph
[18:18] * blank (~Joppe4899@exit0.radia.tor-relays.net) Quit ()
[18:20] * mhack (~mhack@nat-pool-bos-t.redhat.com) Quit (Quit: I'm outta here!)
[18:20] * blizzow (~jburns@50-243-148-102-static.hfc.comcastbusiness.net) has joined #ceph
[18:21] * Be-El (~blinke@nat-router.computational.bio.uni-giessen.de) Quit (Quit: Leaving.)
[18:24] * lincolnb (~lincoln@c-71-57-68-189.hsd1.il.comcast.net) Quit (Ping timeout: 480 seconds)
[18:31] * vicente (~vicente@1-161-184-59.dynamic.hinet.net) has joined #ceph
[18:35] * mhack (~mhack@nat-pool-bos-t.redhat.com) has joined #ceph
[18:38] * mhack (~mhack@nat-pool-bos-t.redhat.com) Quit (Remote host closed the connection)
[18:39] * vicente (~vicente@1-161-184-59.dynamic.hinet.net) Quit (Ping timeout: 480 seconds)
[18:43] * ntpttr_ (~ntpttr@134.134.139.77) Quit (Remote host closed the connection)
[18:43] * ntpttr_ (~ntpttr@134.134.139.77) has joined #ceph
[18:45] * mattch1 (~mattch@w5430.see.ed.ac.uk) Quit (Ping timeout: 480 seconds)
[18:45] * gong (~oftc-webi@89-160-187-133.du.xdsl.is) Quit (Quit: Page closed)
[18:45] * xinli (~charleyst@32.97.110.54) Quit (Ping timeout: 480 seconds)
[18:55] * baotiao (~baotiao@43.255.178.177) Quit (Quit: baotiao)
[18:55] * DanFoster (~Daniel@2a00:1ee0:3:1337:1000:2cd1:f412:5d37) Quit (Quit: Leaving)
[18:57] * baotiao (~baotiao@43.255.178.177) has joined #ceph
[18:59] * mykola (~Mikolaj@91.245.74.212) has joined #ceph
[19:00] * mnc (~mnc@c-50-137-214-131.hsd1.mn.comcast.net) has joined #ceph
[19:03] * cathode (~cathode@50-232-215-114-static.hfc.comcastbusiness.net) has joined #ceph
[19:04] * kristen (~kristen@134.134.139.76) has joined #ceph
[19:05] * ade (~abradshaw@ip9234f2b8.dynamic.kabel-deutschland.de) has joined #ceph
[19:11] * derjohn_mob (~aj@88.128.80.147) has joined #ceph
[19:12] * wes_dillingham (~wes_dilli@140.247.242.44) Quit (Quit: wes_dillingham)
[19:15] * foxxx0 (~fox@2a01:4f8:171:1fd3:0:2:0:2) Quit (Quit: WeeChat 1.5)
[19:15] * foxxx0 (~fox@valhalla.nano-srv.net) has joined #ceph
[19:20] * KindOne_ (kindone@h252.172.16.98.dynamic.ip.windstream.net) has joined #ceph
[19:23] * andreww (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[19:26] * KindOne (kindone@0001a7db.user.oftc.net) Quit (Ping timeout: 480 seconds)
[19:26] * KindOne_ is now known as KindOne
[19:28] * baotiao (~baotiao@43.255.178.177) Quit (Quit: baotiao)
[19:32] * vicente (~vicente@1-161-184-59.dynamic.hinet.net) has joined #ceph
[19:35] * andreww (~xarses@64.124.158.3) has joined #ceph
[19:37] * AdamOfDC949 (~oftc-webi@c-50-129-97-100.hsd1.il.comcast.net) Quit (Quit: Page closed)
[19:40] * vicente (~vicente@1-161-184-59.dynamic.hinet.net) Quit (Ping timeout: 480 seconds)
[19:43] * BrianA (~BrianA@fw-rw.shutterfly.com) has joined #ceph
[19:47] * wes_dillingham (~wes_dilli@140.247.242.44) has joined #ceph
[19:53] * stiopa (~stiopa@cpc73832-dals21-2-0-cust453.20-2.cable.virginm.net) has joined #ceph
[19:56] * blizzow (~jburns@50-243-148-102-static.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[19:57] * ntpttr__ (~ntpttr@134.134.139.77) has joined #ceph
[19:57] * ntpttr_ (~ntpttr@134.134.139.77) Quit (Remote host closed the connection)
[20:06] * blizzow (~jburns@50-243-148-102-static.hfc.comcastbusiness.net) has joined #ceph
[20:07] * derjohn_mob (~aj@88.128.80.147) Quit (Ping timeout: 480 seconds)
[20:08] * dgurtner (~dgurtner@84-73-130-19.dclient.hispeed.ch) Quit (Ping timeout: 480 seconds)
[20:11] * jarrpa (~jarrpa@167.220.56.167) has joined #ceph
[20:22] * davidzlap (~Adium@2605:e000:1313:8003:5bf:bfa9:b2f8:c948) has joined #ceph
[20:24] * cathode (~cathode@50-232-215-114-static.hfc.comcastbusiness.net) Quit (Quit: Leaving)
[20:30] * xinli (~charleyst@32.97.110.55) has joined #ceph
[20:31] * Jeffrey4l__ (~Jeffrey@110.252.57.207) Quit (Ping timeout: 480 seconds)
[20:37] * Hemanth (~hkumar_@103.228.221.140) Quit (Ping timeout: 480 seconds)
[20:44] * achieva (~joonsung@foresee.postech.ac.kr) has joined #ceph
[20:45] <achieva> Hi, I'm trying to find a way to use rbd cache (supported by librbd).
[20:46] <achieva> At the first time, I use "rbd map" to import an image into block device, however, I realized that "rbd map" uses krbd whose caching policy is similar to page cache, not rbd cache.
[20:47] <achieva> So, now, I try to find a way to export block device via librbd, not krbd
[20:47] <achieva> Is there any way to do that?
[20:47] <achieva> export -> import
[20:48] * salwasser (~Adium@72.246.3.14) Quit (Quit: Leaving.)
[20:57] * ntpttr_ (~ntpttr@192.55.54.36) has joined #ceph
[20:57] * ntpttr__ (~ntpttr@134.134.139.77) Quit (Remote host closed the connection)
[21:11] <mlovell> rbd-nbd maybe
[21:19] * fridim (~fridim@56-198-190-109.dsl.ovh.fr) Quit (Ping timeout: 480 seconds)
[21:21] * hoonetorg (~hoonetorg@77.119.226.254.static.drei.at) Quit (Ping timeout: 480 seconds)
[21:25] * wjw-freebsd (~wjw@smtp.digiware.nl) has joined #ceph
[21:26] * sudocat (~dibarra@192.185.1.20) Quit (Ping timeout: 480 seconds)
[21:31] * hoonetorg (~hoonetorg@77.119.226.254.static.drei.at) has joined #ceph
[21:36] * bniver (~bniver@71-9-144-29.static.oxfr.ma.charter.com) Quit (Remote host closed the connection)
[21:36] * nils_ (~nils_@doomstreet.collins.kg) Quit (Quit: This computer has gone to sleep)
[21:38] * Hemanth (~hkumar_@103.228.221.140) has joined #ceph
[21:42] * Hemanth (~hkumar_@103.228.221.140) Quit ()
[21:46] * LongyanG (~long@15255.s.t4vps.eu) Quit (Ping timeout: 480 seconds)
[21:52] <blizzow> I'm adding an OSD node with 4 OSDs in it to my cluster. Adding all 4 OSDs at once really affected performance, so I decided to add one drive at a time with ceph-deploy. The recovery with one drive is so darn slow. I know there is a number or running recovery threads toggle I can mess with. Is there a bandwidth specific toggle?
[21:54] * mnc (~mnc@c-50-137-214-131.hsd1.mn.comcast.net) Quit (Quit: Leaving)
[22:02] <mlovell> setting osd_max_backfills on the osds can limit how much is done at once. it defaults to 10. i've found that can dramatically impact performance. i usually keep it at 1 or 2. it defines how many PGs can be backfilling at once on an OSD.
[22:04] <mlovell> osd_backfill_scan_max and osd_backfill_scan_min might also help but i haven't spent much time adjusting them.
[22:07] <blizzow> Last week, I had to drop down my max_backfills to 1 because adding an OSD node brought my entire (previously healthy) cluster to it's knees with the default of 10.
[22:09] * mykola (~Mikolaj@91.245.74.212) Quit (Quit: away)
[22:09] * xinli (~charleyst@32.97.110.55) Quit ()
[22:10] * LongyanG (~long@15255.s.t4vps.eu) has joined #ceph
[22:12] * xENO_1 (~dontron@108.61.123.73) has joined #ceph
[22:16] * derjohn_mob (~aj@x4db06885.dyn.telefonica.de) has joined #ceph
[22:23] <mlovell> those are the only options i know of for controlling the speed of backfill.
[22:23] * Racpatel (~Racpatel@2601:87:3:31e3::4d2a) Quit (Quit: Leaving)
[22:24] <mlovell> it would be nice if there is/was a setting for limiting either the bytes/sec or objects/minute of backfill.
[22:24] <mlovell> but i don't know of any
[22:26] * georgem (~Adium@206.108.127.16) Quit (Ping timeout: 480 seconds)
[22:28] * wes_dillingham (~wes_dilli@140.247.242.44) Quit (Ping timeout: 480 seconds)
[22:29] * wak-work (~wak-work@2620:15c:2c5:3:a4c5:19d8:d655:a7ea) Quit (Ping timeout: 480 seconds)
[22:32] * davidzlap1 (~Adium@cpe-172-91-154-245.socal.res.rr.com) has joined #ceph
[22:33] <FidoNet> hi
[22:33] * newbie (~kvirc@host217-114-156-249.pppoe.mark-itt.net) Quit (Ping timeout: 480 seconds)
[22:33] <FidoNet> I???m having some problems with ceph_fs and I presume permissions ???. smacks of http://tracker.ceph.com/issues/13730
[22:35] * davidzlap2 (~Adium@cpe-172-91-154-245.socal.res.rr.com) has joined #ceph
[22:35] * davidzlap1 (~Adium@cpe-172-91-154-245.socal.res.rr.com) Quit (Read error: Connection reset by peer)
[22:38] * wak-work (~wak-work@2620:15c:2c5:3:ed41:8500:8d0:fe8a) has joined #ceph
[22:40] * davidzlap (~Adium@2605:e000:1313:8003:5bf:bfa9:b2f8:c948) Quit (Ping timeout: 480 seconds)
[22:42] * xENO_1 (~dontron@108.61.123.73) Quit ()
[22:46] * sudocat (~dibarra@104-188-116-197.lightspeed.hstntx.sbcglobal.net) has joined #ceph
[22:55] * Racpatel (~Racpatel@2601:87:3:31e3::4d2a) has joined #ceph
[22:55] * sudocat (~dibarra@104-188-116-197.lightspeed.hstntx.sbcglobal.net) Quit (Ping timeout: 480 seconds)
[23:00] * ade (~abradshaw@ip9234f2b8.dynamic.kabel-deutschland.de) Quit (Quit: Too sexy for his shirt)
[23:03] * davidzlap (~Adium@cpe-172-91-154-245.socal.res.rr.com) has joined #ceph
[23:03] * mattbenjamin1 (~mbenjamin@12.118.3.106) Quit (Quit: Leaving.)
[23:06] * sudocat (~dibarra@2602:306:8bc7:4c50:2d7d:a0ed:89ab:8683) has joined #ceph
[23:06] * davidzlap2 (~Adium@cpe-172-91-154-245.socal.res.rr.com) Quit (Ping timeout: 480 seconds)
[23:09] * davidzlap1 (~Adium@2605:e000:1313:8003:513a:75bc:802b:182b) has joined #ceph
[23:10] * davidzlap (~Adium@cpe-172-91-154-245.socal.res.rr.com) Quit (Read error: Connection reset by peer)
[23:10] * ntpttr_ (~ntpttr@192.55.54.36) Quit (Remote host closed the connection)
[23:10] * ntpttr_ (~ntpttr@134.134.139.77) has joined #ceph
[23:18] * georgem (~Adium@24.114.53.188) has joined #ceph
[23:23] * TMM (~hp@dhcp-077-248-009-229.chello.nl) has joined #ceph
[23:24] * fdmanana (~fdmanana@2001:8a0:6e0c:6601:1cec:4075:a3a:1bce) Quit (Ping timeout: 480 seconds)
[23:26] * georgem (~Adium@24.114.53.188) has left #ceph
[23:37] <blizzow> I have 1 OSD node with 8x4TB 7200RPM drives in it. 2 OSD nodes with 4x4TB 7200RPM drives Then I have 8 OSD nodes with 3x1TB 7200 RPM drives and 1x1TB crappy(not suitable for a journal) SSD. I have one pool for VMs that will be about 15-20TB for VM storage. I want to use the rest of my store for archival storage (using rgw like an S3 store). What should my PGs and crush map look like?
[23:43] * BrianA (~BrianA@fw-rw.shutterfly.com) Quit (Read error: Connection reset by peer)
[23:44] <blizzow> I have a Dell PERC H700 controller in front of a some 7200 RPM disks on a hypervisor. I'm wondering what kind of caching setup I should be using to provide a VM the fastest disk access possible to the disks. Should I enable writeback on the raid controller and turn off caching on the VM? Should I be using writeback on both? Should I be using virtio or virtio-scsi?
[23:45] * rakeshgm (~rakesh@38.140.108.5) has joined #ceph
[23:48] * wes_dillingham (~wes_dilli@209-6-222-74.c3-0.hdp-ubr1.sbo-hdp.ma.cable.rcn.com) has joined #ceph
[23:50] * wes_dillingham (~wes_dilli@209-6-222-74.c3-0.hdp-ubr1.sbo-hdp.ma.cable.rcn.com) Quit ()
[23:50] * axion_joey_ (~oftc-webi@108.47.170.18) has joined #ceph

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.