#ceph IRC Log

Index

IRC Log for 2015-12-09

Timestamps are in GMT/BST.

[0:00] <TMM> Anticimex, I would agree with you that the existing puppet manifests for ceph are pretty bad. I doubt mine will solve your particular problem either but they work well for me.
[0:00] * brad_mssw (~brad@66.129.88.50) Quit (Quit: Leaving)
[0:01] <TMM> you really just instantiate a ceph class and the rest is all hiera
[0:02] * enax (~enax@94-21-125-92.pool.digikabel.hu) has joined #ceph
[0:02] <TMM> you also have to manually import the keys before you start your osds
[0:02] * enax (~enax@94-21-125-92.pool.digikabel.hu) has left #ceph
[0:03] <TMM> this setup assumes you've pregenerated them and put them in hiera
[0:03] <TMM> preferably encrypted :P
[0:03] <TMM> so after the monitors are up you just ceph auth import them
[0:06] <TMM> Anticimex, http://pastebin.com/yayQTiM7
[0:06] <TMM> that's the abridged version of the setup after the first puppet run
[0:08] <fedgoat> hmm, anyone point me to the easiest way to recover a failed journal --flush-journal failed with a failed journal dump also..
[0:09] <TMM> fedgoat, you can just take the osd out of the cluster and create a new journal
[0:10] <fedgoat> well the problem is I'm having some PG peering problems on separate OSD's not sure this OSD doesn't contain any history, so reluctant to remove this odd and re-create
[0:10] <TMM> you should probably see if the pg has enough copies first then :)
[0:11] <TMM> fedgoat, http://docs.ceph.com/docs/master/rados/troubleshooting/troubleshooting-pg/ this has the info you need to make that determination
[0:11] <fedgoat> yea..i have about 18 pig's stuck peering active
[0:12] <TMM> that's not necessarily a problem
[0:12] <fedgoat> I'm recovering from a very strange issue here unfortunately
[0:14] <fedgoat> 77pgs stuck inactive and 80 stuck unclean I've had to reduce the min_size on the pools also down to 1 and was able to get some to Peer..whoever architected this decided that the recommended 50-100 pg's per pool weren't acceptable and we're at 1000, was seeing some super high CPU usages killing the cluster on the ceph-osd process, and deadlock on these 3 nodes, and have had to turn all the threading down just to get the clus
[0:14] <fedgoat> terinto a recovery state
[0:15] <fedgoat> its almost like we hit some sort of race condition, where heartbeat was failing between osds due to such high CPU possibly caused by the super high number of PG's per OSD
[0:19] * georgem (~Adium@75-119-226-89.dsl.teksavvy.com) Quit (Quit: Leaving.)
[0:20] * samx (~Adium@12.206.204.58) Quit (Quit: Leaving.)
[0:20] * KaneK (~kane@12.206.204.58) Quit (Quit: KaneK)
[0:22] * allaok (~allaok@ARennes-658-1-83-253.w90-32.abo.wanadoo.fr) Quit (Quit: Leaving.)
[0:22] * johnavp1989 (~jpetrini@pool-100-14-5-21.phlapa.fios.verizon.net) has joined #ceph
[0:22] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[0:22] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[0:22] * johnavp1989 (~jpetrini@pool-100-14-5-21.phlapa.fios.verizon.net) has left #ceph
[0:27] * jdillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) Quit (Quit: jdillaman)
[0:31] * Tonux (~Eric@tor2r.ins.tor.net.eu.org) has joined #ceph
[0:35] * Guest684 (~herrsergi@200.77.224.239) Quit (Ping timeout: 480 seconds)
[0:36] * cathode (~cathode@50-198-166-81-static.hfc.comcastbusiness.net) has joined #ceph
[0:36] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[0:36] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[0:36] <cathode> hi guys. i'mt trying to find some information on whether it's acceptable to dual-purpose ceph nodes as hypervisor hosts. like, a cluster of 'all-in-one' machines
[0:37] <cathode> lets say 5 identical systems with some local disks and good CPUs and lots of ram. run monitor nodes on 3, OSD on all five, and some VMs on all five
[0:38] * Exchizz (~kvirc@0x3ec6d532.inet.dsl.telianet.dk) Quit (Ping timeout: 480 seconds)
[0:42] * KaneK (~kane@12.206.204.58) has joined #ceph
[0:47] <fedgoat> hrm..ever have a cluster have negative objects degraded.
[0:47] * garphy is now known as garphy`aw
[0:55] * mattbenjamin (~mbenjamin@aa2.linuxbox.com) Quit (Quit: Leaving.)
[1:01] * Tonux (~Eric@6YRAABF50.tor-irc.dnsbl.oftc.net) Quit ()
[1:01] * Pirate (~Sigma@195-154-191-67.rev.poneytelecom.eu) has joined #ceph
[1:01] * stiopa (~stiopa@cpc73828-dals21-2-0-cust630.20-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[1:07] * shawniverson (~shawniver@208.38.236.8) Quit (Remote host closed the connection)
[1:10] * lcurtis_ (~lcurtis@47.19.105.250) Quit (Remote host closed the connection)
[1:11] <cathode> kinda dead in here..
[1:12] <fedgoat> fixed it...HEALTH_OK after 5 days nice
[1:12] <fedgoat> they never really say what happens if you overload PG's per OSD yea..don't do it...
[1:15] * tacticus (~tacticus@kca.id.au) Quit (Quit: WeeChat 1.1.1)
[1:15] * tacticus (~tacticus@v6.kca.id.au) has joined #ceph
[1:16] <cathode> so... any feedback on building a small cluster of all-in-one nodes?
[1:17] * KaZeR (~KaZeR@64.201.252.132) Quit (Quit: Leaving...)
[1:22] <mfa298> cathode: I'm not sure it's something most people would recommend. It might work ok when the cluster is healthy, but have various issues if there's lots of recovery io happening
[1:23] <mfa298> I think there were also warnings about having rbd volumes and/or cephfs mounted on the same hosts as the OSD processes
[1:23] * ndevos (~ndevos@nat-pool-ams2-5.redhat.com) has joined #ceph
[1:23] * ndevos_ (~ndevos@nat-pool-ams2-5.redhat.com) Quit (Read error: Connection reset by peer)
[1:24] * EinstCra_ (~EinstCraz@218.69.65.22) Quit (Remote host closed the connection)
[1:24] * EinstCrazy (~EinstCraz@218.69.65.22) has joined #ceph
[1:25] <cathode> ok...hmm
[1:25] <cathode> so ideally the storage is separate from the compute nodes?
[1:29] <Thunderbird> ceph needs some cpu, there are guidelines, depending on your workload it could work
[1:29] <Thunderbird> you could always do some cpu pinning or something to that effect
[1:29] * toMeloos (~toMeloos@53568B3D.cm-6-7c.dynamic.ziggo.nl) Quit (Ping timeout: 480 seconds)
[1:31] * Pirate (~Sigma@6YRAABF6W.tor-irc.dnsbl.oftc.net) Quit ()
[1:31] * Sophie1 (~Plesioth@4Z9AABVJK.tor-irc.dnsbl.oftc.net) has joined #ceph
[1:31] * jclm (~jclm@ip68-108-16-17.lv.lv.cox.net) Quit (Quit: Leaving.)
[1:32] * EinstCrazy (~EinstCraz@218.69.65.22) Quit (Ping timeout: 480 seconds)
[1:33] <cathode> hmmm ok
[1:34] <cathode> my boss wants $$$$$ reliability, but only wants to pay $ for it
[1:34] * wyang (~wyang@46.21.158.66) has joined #ceph
[1:35] <cathode> so i set up an active-passive pair of freebsd machines using ZFS and HAST+CARP to replicate data in real-time between the two nodes. it's a bit finnicky but we've utilized the failover a couple times already. so it works okay. he still wants something better which has me looking into Ceph
[1:35] <mfa298> you might get away with it with well specced servers and a small number of osds. although checkout any issues with rbd/cephfs like that if youre planning on going that route
[1:37] <cathode> ok. it looks like most deployments use one OSD per spindle, but since there's a memory overhead for OSDs would I be better off doing like either raid10 or raid1 via zfs or btrfs and putting OSDs on top of that?
[1:40] <mfa298> I think the only really recommended fs is zfs still, btrfs in theory has better features, but I think there are some issues (performance over time) which mean its not generally suggested for production systems
[1:40] <mfa298> sorry, I mean xfs
[1:40] * samx (~Adium@12.206.204.58) has joined #ceph
[1:40] <cathode> ok
[1:40] <Thunderbird> in general raid is not recommended, ceph essentially does its own in software
[1:41] <Thunderbird> overhead for OSDs is not so bad
[1:41] <samx> anyone have documentation on the "exclusive-lock" feature of rbd?
[1:41] <Thunderbird> look at the hardware guidelines
[1:41] <cathode> hardware guidelines that i've read are typically for large-scale clusters. I'm trying to scale *down* as much as possible. maybe ceph isn't a good fit?
[1:42] <Thunderbird> the guidelines are simple
[1:42] <Thunderbird> e.g. 1 GHz (physical core) per OSD and 1-2 GB per OSD
[1:42] <mfa298> If some disks is 3-5, and lots of CPU is 2x 12c processes and 256GB ram, then there's probably plenty of capacity to do both
[1:43] <cathode> ok
[1:43] <mfa298> if youre more into 12 osds and 2x 6c processes and 64GB ram then that could struggle with ceph + vms at times
[1:44] * tsg (~tgohad@192.55.54.43) Quit (Remote host closed the connection)
[1:44] <mfa298> part of it depends on what you mean by some and lots.
[1:44] <cathode> i was planning on using 2U chassis w/ 24 x 2.5 bays and 300 or 600GB 10k SAS drives
[1:44] <cathode> for each node
[1:45] <cathode> we don't have space needs, really. i think including snapshots for the last year (taken every 2 hours) we had 3TB of used space on our current backend
[1:45] <Thunderbird> then you indeed lots of cpu & ram
[1:46] <Thunderbird> if you need to cut down on costs 3.5" drives are fine too, you don't need SAS
[1:46] <cathode> alright i am still completely new to this, but to me that seems like ceph is really inefficient when it comes to resources?
[1:46] <Thunderbird> most of the resources are needed during synchronization / recovery
[1:46] <Thunderbird> any data movement system needs quite some resources even NFS servers
[1:47] <Thunderbird> I/O is not free
[1:47] <cathode> i mean we have older-generation (Nehalem/Westmere) CPUs in our current storage nodes and i run LZ4 compression on my ZFS backends and never see more than like 5% cpu usage on a single quad-core
[1:47] <mfa298> idle state it's not too bad, but recovery can lead to lots of data to track which leads to more memory overhead.
[1:50] <mfa298> remember that ceph is trying to keep 3 (as a default) copies of your data on different systems, If a drive or host goes down ceph will automatically try and get to a state where it still has three copies.
[1:51] <cathode> right...
[1:51] <cathode> can't i just run one OSD per physical storage node?
[1:51] <mfa298> if drives go up and down during that process that gives more possibilities of wher ethe data could be stored so there's more maps to look through.
[1:52] <mfa298> one big OSD may make that worse in some ways as there's more data to syncronise if it goes down.
[1:56] <cathode> alright...
[1:56] <cathode> well, thanks for the advice :) i need to see if we can buy some equipment so i can set up a test lab with real hardware
[1:57] * xarses (~xarses@64.124.158.100) Quit (Ping timeout: 480 seconds)
[1:58] <mfa298> test lab, with lots of data in it, try and simulate real workloads on the vms, then start pulling disks (or even a whole node) from ceph and see how it goes.
[1:58] <Thunderbird> cathode, various hardware vendors sell reference ceph boxes, may make sense to just buy those e.g. supermicro
[1:59] <Thunderbird> without having to reinvent yourself and spend weeks on it
[2:01] * Sophie1 (~Plesioth@4Z9AABVJK.tor-irc.dnsbl.oftc.net) Quit ()
[2:01] <fedgoat> we run AIO nodes in production, but don't run CephFS, just block and object Rgw, OSD, mon
[2:03] <fedgoat> wouldn't recommend it if you're running MDS/CephFS...works fine for an RBD backend for glance and cinder for the most part
[2:03] * EinstCrazy (~EinstCraz@111.30.21.47) has joined #ceph
[2:05] * LeaChim (~LeaChim@host86-185-146-193.range86-185.btcentralplus.com) Quit (Ping timeout: 480 seconds)
[2:06] * sudocat (~dibarra@192.185.1.20) Quit (Ping timeout: 480 seconds)
[2:09] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[2:09] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[2:10] * KaneK (~kane@12.206.204.58) Quit (Quit: KaneK)
[2:14] * xarses (~xarses@50.141.34.130) has joined #ceph
[2:14] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[2:14] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[2:16] * linjan (~linjan@176.195.68.96) Quit (Ping timeout: 480 seconds)
[2:17] * samx (~Adium@12.206.204.58) Quit (Ping timeout: 480 seconds)
[2:24] * xarses (~xarses@50.141.34.130) Quit (Ping timeout: 480 seconds)
[2:28] * arsenaali (~Dysgalt@tsn109-201-152-225.dyn.nltelcom.net) has joined #ceph
[2:33] * swami1 (~swami@27.7.168.203) has joined #ceph
[2:36] * wjw-freebsd (~wjw@smtp.digiware.nl) Quit (Ping timeout: 480 seconds)
[2:41] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[2:41] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Remote host closed the connection)
[2:44] * cholcombe (~chris@c-73-180-29-35.hsd1.or.comcast.net) Quit (Ping timeout: 481 seconds)
[2:45] * wushudoin (~wushudoin@2601:646:8201:7769:2ab2:bdff:fe0b:a6ee) Quit (Ping timeout: 480 seconds)
[2:50] * yanzheng (~zhyan@182.139.205.155) has joined #ceph
[2:50] * KaneK (~kane@cpe-172-88-240-14.socal.res.rr.com) has joined #ceph
[2:52] * jdillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) has joined #ceph
[2:53] * swami1 (~swami@27.7.168.203) Quit (Quit: Leaving.)
[2:53] * jdillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) Quit ()
[2:57] * olid1111117 (~olid1982@aftr-185-17-204-188.dynamic.mnet-online.de) Quit (Ping timeout: 480 seconds)
[2:58] * arsenaali (~Dysgalt@6YRAABF8Z.tor-irc.dnsbl.oftc.net) Quit ()
[3:01] * adun153 (~ljtirazon@202.69.177.69) has joined #ceph
[3:02] * zhaochao (~zhaochao@60.206.230.18) has joined #ceph
[3:04] * xarses (~xarses@50.141.34.146) has joined #ceph
[3:06] * linuxkidd (~linuxkidd@72.sub-70-210-225.myvzw.com) Quit (Ping timeout: 481 seconds)
[3:06] * linuxkidd (~linuxkidd@72.sub-70-210-225.myvzw.com) has joined #ceph
[3:07] * naoto (~naotok@2401:bd00:b001:8920:27:131:11:254) has joined #ceph
[3:20] * jdillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) has joined #ceph
[3:21] * Concubidated (~Adium@pool-98-119-93-148.lsanca.fios.verizon.net) Quit (Remote host closed the connection)
[3:24] <Nats__> cathode, yes ceph is a pig when it comes to cpu+memory
[3:34] * Sigma (~Spessu@109.201.143.40) has joined #ceph
[3:37] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[3:37] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[3:37] * naoto_ (~naotok@27.131.11.254) has joined #ceph
[3:38] * Concubidated (~Adium@pool-98-119-93-148.lsanca.fios.verizon.net) has joined #ceph
[3:38] * vasu (~vasu@c-73-231-60-138.hsd1.ca.comcast.net) Quit (Quit: Leaving)
[3:39] * mhack (~mhack@66-168-117-78.dhcp.oxfr.ma.charter.com) Quit (Remote host closed the connection)
[3:43] * naoto (~naotok@2401:bd00:b001:8920:27:131:11:254) Quit (Ping timeout: 480 seconds)
[3:44] * xarses_ (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) has joined #ceph
[3:50] * xarses (~xarses@50.141.34.146) Quit (Ping timeout: 480 seconds)
[3:51] * moore (~moore@64.202.160.233) Quit (Remote host closed the connection)
[4:01] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[4:01] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[4:04] * dlan (~dennis@116.228.88.131) Quit (Quit: leaving)
[4:04] * Sigma (~Spessu@4Z9AABVNC.tor-irc.dnsbl.oftc.net) Quit ()
[4:05] * dlan (~dennis@116.228.88.131) has joined #ceph
[4:07] * joshd1 (~jdurgin@68-119-140-18.dhcp.ahvl.nc.charter.com) Quit (Quit: Leaving.)
[4:14] * wulb (~wulb@123.58.191.68) has joined #ceph
[4:15] <wulb> Hi, any sugesstions about incomplete pg?
[4:17] <badone> wulb: what version and what can you tell us about it?
[4:20] <wulb> 0.80.11
[4:21] <badone> wulb: okay, so what can you tell us about it?
[4:28] <wulb> hmm.., afraid it's a long story.. The steps is: {1. re-deploy host1; 2. re-deploy host2, has a active+degrade + remapped pg, one unfound object; 3. re-deploy host3, at the same time, host 4 has been restarted.., but finally, also one pg abnormal; 5. re-deploy host5, two pg abnormal; 6. i follow the ceph troubleshooting of pg do command: ceph pg 13.b5 mark_unfound_lost revert, then one pg become
[4:28] <wulb> ok, another pg is incomplete}
[4:29] <badone> wulb: what does a query of that pg look like?
[4:30] <badone> wulb: are all your OSDs upa nd in?
[4:33] * dmatson1 (~david.mat@192.41.52.12) has joined #ceph
[4:34] * shaunm (~shaunm@208.102.161.229) Quit (Ping timeout: 480 seconds)
[4:34] <wulb> yeah, all osd up and in, but you know, we has two osds on one SSD before re-deploy, now it's one osd per SSD.
[4:34] <badone> wulb: what does a query of that pg look like?
[4:34] <wulb> query is so long, let me try to paste here
[4:35] <badone> use a pastebin
[4:37] * dmatson (~david.mat@192.41.52.12) Quit (Ping timeout: 480 seconds)
[4:40] * bloatyfloat_ (~bloatyflo@46.37.172.253.srvlist.ukfast.net) has joined #ceph
[4:43] * cholcombe (~chris@c-73-180-29-35.hsd1.or.comcast.net) has joined #ceph
[4:45] <wulb> good idea!
[4:45] <wulb> http://pastebin.com/SSYDhcqA
[4:47] * bloatyfloat (~bloatyflo@46.37.172.253.srvlist.ukfast.net) Quit (Ping timeout: 480 seconds)
[4:47] * bloatyfloat_ is now known as bloatyfloat
[4:48] <badone> wulb: what is the story with OSD 41?
[4:52] <wulb> oh, if my memroy no wrong, first probing_osds is [45, 15, 47, 41], then i out the 41, so the 41 went to the "down_osds_we_would_probe"
[4:53] <badone> wulb: you just said " yeah, all osd up and in"
[4:54] <wulb> oh, but 41 is not existed now
[4:54] <wulb> -2 3.96 host 10-180-0-34
[4:54] <wulb> 0 0.72 osd.0 up 1
[4:54] <wulb> 1 0.72 osd.1 up 1
[4:54] <wulb> 7 0.54 osd.7 up 1
[4:54] <wulb> 42 0.54 osd.42 up 1
[4:54] <wulb> 43 0.72 osd.43 up 1
[4:54] <wulb> 45 0.72 osd.45 up 1
[4:54] <wulb> -3 4.32 host 10-180-0-30
[4:54] <wulb> 3 0.72 osd.3 up 1
[4:54] <wulb> 4 0.72 osd.4 up 1
[4:54] <wulb> 25 0.72 osd.25 up 1
[4:54] <wulb> 44 0.72 osd.44 up 1
[4:54] <wulb> 46 0.72 osd.46 up 1
[4:54] <wulb> 47 0.72 osd.47 up 1
[4:54] <wulb> -4 3.6 host 10-180-0-37
[4:54] <wulb> 10 0.54 osd.10 up 1
[4:54] <wulb> 11 0.54 osd.11 up 1
[4:54] <wulb> 12 0.54 osd.12 up 1
[4:54] <wulb> 13 0.54 osd.13 up 1
[4:54] <wulb> 21 0.72 osd.21 up 1
[4:55] <wulb> 27 0.72 osd.27 up 1
[4:55] <wulb> -5 3.96 host 10-180-0-46
[4:55] <wulb> 8 0.72 osd.8 up 1
[4:55] <wulb> 17 0.72 osd.17 up 1
[4:55] <wulb> 18 0.72 osd.18 up 1
[4:55] <wulb> 19 0.72 osd.19 up 1
[4:55] <wulb> 20 0.54 osd.20 up 1
[4:55] <wulb> 22 0.54 osd.22 up 1
[4:55] <wulb> -6 4.32 host 10-180-0-120
[4:55] <wulb> 9 0.72 osd.9 up 1
[4:55] <wulb> 14 0.72 osd.14 up 1
[4:55] <wulb> 15 0.72 osd.15 up 1
[4:55] <wulb> 16 0.72 osd.16 up 1
[4:55] <wulb> 23 0.72 osd.23 up 1
[4:55] <wulb> 28 0.72 osd.28 up 1
[4:55] <wulb> -7 4.32 host 10-180-0-121
[4:55] <wulb> 30 0.72 osd.30 up 1
[4:55] <wulb> 31 0.72 osd.31 up 1
[4:55] <wulb> 33 0.72 osd.33 up 1
[4:55] <wulb> 34 0.72 osd.34 up 1
[4:55] <wulb> 32 0.72 osd.32 up 1
[4:55] <wulb> 29 0.72 osd.29 up 1
[4:55] <wulb> -8 0 host 10-180-0-36
[4:55] <wulb> -9 0 host 10-180-0-31
[4:55] <wulb> -10 0 host 10-180-0-35
[4:55] <wulb> 2 0 osd.2 down 0
[4:55] <wulb> 5 0 osd.5 down 0
[4:55] <wulb> 6 0 osd.6 down 0
[4:55] <wulb> 24 0 osd.24 down 0
[4:55] <wulb> 26 0 osd.26 down 0
[4:55] <wulb> 69 0 osd.69 down 0
[4:55] <wulb> sorry, not good paste here...
[4:55] <badone> no, it isn't
[4:56] <mfa298> wulb: when you re-deployed each host did you let the cluster get to health ok, with all pgs listed as active+clean before doing the next host.
[4:56] <badone> I would say not
[4:56] <mfa298> that's what I would suspect
[4:57] <wulb> mfa298: yeah, from my steps, after my re-deploy host2, one pg abnormal.
[4:57] <wulb> badone: what do you mean?
[4:58] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[4:58] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[4:58] * brutuscat (~brutuscat@105.34.133.37.dynamic.jazztel.es) Quit (Remote host closed the connection)
[4:58] <badone> wulb: looks like 41 probably had data on it that is now gone (if 41 is gone)
[4:58] <wulb> badone: yeah, make sense
[4:59] <wulb> badone: 41 is really gone.
[4:59] <mfa298> it would probably have een sensible to have stopped re-deploying hosts as soon as and error occured
[4:59] <badone> wulb: what size is pool 13
[4:59] <badone> ?
[5:00] <badone> right
[5:02] * kanagaraj (~kanagaraj@121.244.87.117) has joined #ceph
[5:02] * adun153 (~ljtirazon@202.69.177.69) Quit (Quit: Leaving)
[5:03] <wulb> mfa298: yeah, it's a accident, we use a common computer with others. Then reboot it suddenly
[5:03] <wulb> badone: sudo ceph osd pool get vms size
[5:03] <wulb> size: 3
[5:03] * Concubidated (~Adium@pool-98-119-93-148.lsanca.fios.verizon.net) Quit (Remote host closed the connection)
[5:03] <wulb> badone: try to set min_size to 1, no works.
[5:05] <badone> wulb: I imagine you've tried to repair the pg?
[5:06] <wulb> badone: Yeah, repair, deep-scrub, receate..
[5:06] <wulb> But, useless
[5:08] <wulb> Another performance question: 1. run fio on six rbd devices. 2. stop one osd for 10 seconds and start again. 3. after recovery started, the performance drop down from 7k to 1k for about 10 seconds. Is it normal?
[5:08] <wulb> Version is also 0.80.11
[5:08] * dmatson (~david.mat@192.41.52.12) has joined #ceph
[5:10] <wulb> 6 OSD servers with 6 "intel 3500 SSD"
[5:11] * mollstam (~Esge@tsn109-201-152-225.dyn.nltelcom.net) has joined #ceph
[5:12] <wulb> The performance cut down so much is unacceptable in our production. I'm not sure if it's normal for ceph or it improve on the version hammer?
[5:13] * dmatson1 (~david.mat@192.41.52.12) Quit (Ping timeout: 480 seconds)
[5:14] <mfa298> you may be able to tune how much recovery happens by default. But I don't know when those values appeared (so may not be in 0.80)
[5:15] * cholcombe (~chris@c-73-180-29-35.hsd1.or.comcast.net) Quit (Ping timeout: 480 seconds)
[5:17] <wulb> you mean: osd_recovery_max_active?
[5:18] <wulb> mfa298: I have tried osd_recovery_max_active & osd_max_backfills & osd_recovery_max_chunks, all useless
[5:19] <mfa298> those are the main ones
[5:19] <badone> wulb: try hammer then and see how it goes
[5:20] <mfa298> setting osd_recovery_max_active & osd_max_backfills both to 1 should reduce recovery impact
[5:20] <wulb> badone: So, you don't have conflict this "performance down" problem in hammer?
[5:21] <wulb> mfa298: I already done that, but still more than 10 seconds performance down.
[5:21] <badone> wulb: I'm saying hammer would be worth a try
[5:22] <badone> I know there are recovery performance improvements in hammer
[5:23] * jdillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) Quit (Quit: jdillaman)
[5:23] <wulb> badone: Hmm..ok
[5:27] * krypto (~krypto@65.115.222.48) has joined #ceph
[5:29] * nhm (~nhm@c-50-171-139-246.hsd1.mn.comcast.net) Quit (Ping timeout: 480 seconds)
[5:30] * Vacuum_ (~Vacuum@88.130.211.108) has joined #ceph
[5:31] * dmatson1 (~david.mat@192.41.52.12) has joined #ceph
[5:34] * dmatson (~david.mat@192.41.52.12) Quit (Ping timeout: 480 seconds)
[5:37] * Vacuum__ (~Vacuum@88.130.214.231) Quit (Ping timeout: 480 seconds)
[5:40] * jith (~chatzilla@14.139.180.40) Quit (Quit: ChatZilla 0.9.92 [Firefox 38.0/20150511103818])
[5:40] * Concubidated (~Adium@pool-98-119-93-148.lsanca.fios.verizon.net) has joined #ceph
[5:41] * mollstam (~Esge@4Z9AABVQI.tor-irc.dnsbl.oftc.net) Quit ()
[5:50] * swami1 (~swami@49.44.57.245) has joined #ceph
[5:53] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[5:53] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[5:54] * swami2 (~swami@49.44.57.245) has joined #ceph
[5:54] * fedgoat (~theengine@45-31-177-36.lightspeed.austtx.sbcglobal.net) Quit (Quit: This computer has gone to sleep)
[5:58] * swami1 (~swami@49.44.57.245) Quit (Ping timeout: 480 seconds)
[5:59] * brutuscat (~brutuscat@105.34.133.37.dynamic.jazztel.es) has joined #ceph
[6:10] * theanalyst__ (~abhishekl@223.237.248.15) has joined #ceph
[6:11] * rdas (~rdas@121.244.87.116) has joined #ceph
[6:13] * theanalyst__ (~abhishekl@223.237.248.15) Quit ()
[6:20] * vikhyat (~vumrao@121.244.87.116) has joined #ceph
[6:36] * tomc (~tom.chris@67-2-196-156.slkc.qwest.net) has joined #ceph
[6:36] <tomc> I'm trying to post logs for a bug, and it appears ceph-post-file doesn't work anymore? Is there a recommended way to get 200MB+ of log files associated with a bug in tracker these days?
[6:41] * o0c (~o0c@chris.any.mx) Quit (Remote host closed the connection)
[6:42] * o0c (~o0c@chris.any.mx) has joined #ceph
[6:44] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[6:44] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[6:44] * tomc (~tom.chris@67-2-196-156.slkc.qwest.net) has left #ceph
[6:56] * fcape (~fcape@107-220-57-73.lightspeed.austtx.sbcglobal.net) Quit (Ping timeout: 480 seconds)
[7:02] * ade (~abradshaw@dslb-188-103-023-049.188.103.pools.vodafone-ip.de) has joined #ceph
[7:06] * karnan (~karnan@106.206.129.141) has joined #ceph
[7:21] * samx (~Adium@cpe-172-90-97-62.socal.res.rr.com) has joined #ceph
[7:23] <gregsfortytwo> ugh, ceph-post-file will work again in a few days
[7:24] <gregsfortytwo> another thing that's temporarily broken with the lab being physically moved :(
[7:26] * logan (~a@216.144.251.246) Quit (Ping timeout: 480 seconds)
[7:27] * zaitcev (~zaitcev@c-50-130-189-82.hsd1.nm.comcast.net) Quit (Quit: Bye)
[7:32] * overclk (~vshankar@121.244.87.117) has joined #ceph
[7:32] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[7:32] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[7:32] * theanalyst__ (~abhishekl@223.237.248.15) has joined #ceph
[7:36] * theanalyst__ (~abhishekl@223.237.248.15) Quit ()
[7:41] * karnan (~karnan@106.206.129.141) Quit (Ping timeout: 480 seconds)
[7:43] * karnan (~karnan@121.244.87.117) has joined #ceph
[7:46] * reed (~reed@75-101-54-18.dsl.static.fusionbroadband.com) Quit (Quit: Ex-Chat)
[7:49] * davidzlap1 (~Adium@2605:e000:1313:8003:9576:217c:26b8:8622) has joined #ceph
[7:49] * davidzlap (~Adium@2605:e000:1313:8003:9576:217c:26b8:8622) Quit (Read error: Connection reset by peer)
[7:49] * bvi (~bastiaan@185.56.32.1) has joined #ceph
[7:56] * KaneK (~kane@cpe-172-88-240-14.socal.res.rr.com) Quit (Quit: KaneK)
[7:57] * KaneK (~kane@cpe-172-88-240-14.socal.res.rr.com) has joined #ceph
[7:59] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[7:59] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[7:59] * KaneK (~kane@cpe-172-88-240-14.socal.res.rr.com) Quit ()
[8:00] * KaneK (~kane@cpe-172-88-240-14.socal.res.rr.com) has joined #ceph
[8:04] * KaneK (~kane@cpe-172-88-240-14.socal.res.rr.com) Quit ()
[8:07] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[8:09] * enax (~enax@hq.ezit.hu) has joined #ceph
[8:10] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[8:10] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[8:19] * MrBy (~MrBy@85.115.23.2) Quit (Remote host closed the connection)
[8:20] * jasuarez (~jasuarez@38.Red-81-38-77.dynamicIP.rima-tde.net) has joined #ceph
[8:22] * brutuscat (~brutuscat@105.34.133.37.dynamic.jazztel.es) Quit (Ping timeout: 480 seconds)
[8:22] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[8:22] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[8:22] * swami1 (~swami@49.44.57.245) has joined #ceph
[8:26] * b0e (~aledermue@213.95.25.82) has joined #ceph
[8:26] * dugravot6 (~dugravot6@dn-infra-04.lionnois.univ-lorraine.fr) has joined #ceph
[8:27] * swami2 (~swami@49.44.57.245) Quit (Ping timeout: 480 seconds)
[8:27] * T1w (~jens@node3.survey-it.dk) has joined #ceph
[8:27] * madkiss (~madkiss@2001:6f8:12c3:f00f:8815:aa96:c2a3:316a) has joined #ceph
[8:28] * madkiss (~madkiss@2001:6f8:12c3:f00f:8815:aa96:c2a3:316a) Quit ()
[8:28] * samx (~Adium@cpe-172-90-97-62.socal.res.rr.com) Quit (Quit: Leaving.)
[8:32] * KaneK (~kane@cpe-172-88-240-14.socal.res.rr.com) has joined #ceph
[8:33] * kefu (~kefu@114.92.107.250) has joined #ceph
[8:44] * derjohn_mobi (~aj@p578b6aa1.dip0.t-ipconnect.de) Quit (Remote host closed the connection)
[8:44] * fsimonce (~simon@host28-31-dynamic.30-79-r.retail.telecomitalia.it) has joined #ceph
[8:47] * derjohn_mob (~aj@p578b6aa1.dip0.t-ipconnect.de) has joined #ceph
[9:00] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[9:00] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[9:02] * yanzheng (~zhyan@182.139.205.155) Quit (Quit: ??????)
[9:05] * egonzalez (~egonzalez@238.238.14.62.static.jazztel.es) has joined #ceph
[9:11] * rdas (~rdas@121.244.87.116) Quit (Quit: Leaving)
[9:12] * rdas (~rdas@121.244.87.116) has joined #ceph
[9:12] * yuriw1 (~Adium@2601:645:4380:112c:52a:559f:d49a:9baf) has joined #ceph
[9:12] * rdas (~rdas@121.244.87.116) Quit ()
[9:12] * derjohn_mob (~aj@p578b6aa1.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[9:14] * theanalyst__ (~abhishekl@223.228.132.52) has joined #ceph
[9:17] * derjohn_mob (~aj@p578b6aa1.dip0.t-ipconnect.de) has joined #ceph
[9:18] * toMeloos (~toMeloos@53568B3D.cm-6-7c.dynamic.ziggo.nl) has joined #ceph
[9:19] * yuriw (~Adium@2601:645:4380:112c:383c:64f5:fd5:be7c) Quit (Ping timeout: 480 seconds)
[9:20] * kefu is now known as kefu|afk
[9:21] * kefu|afk (~kefu@114.92.107.250) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[9:24] * rakeshgm (~rakesh@117.99.70.212) has joined #ceph
[9:26] * olid1111117 (~olid1982@aftr-185-17-206-17.dynamic.mnet-online.de) has joined #ceph
[9:27] * mhuang (~mhuang@119.254.120.72) has joined #ceph
[9:28] * thomnico (~thomnico@2a01:e35:8b41:120:2d08:23bb:5f29:f811) has joined #ceph
[9:30] * alexm (~alexm@000128d2.user.oftc.net) Quit (Remote host closed the connection)
[9:30] * alexm (~alexm@chulak.ac.upc.edu) has joined #ceph
[9:30] * rakesh_ (~rakesh@106.51.26.185) has joined #ceph
[9:34] * rakeshgm (~rakesh@117.99.70.212) Quit (Read error: Connection reset by peer)
[9:34] * dgurtner (~dgurtner@178.197.231.21) has joined #ceph
[9:34] * bvi (~bastiaan@185.56.32.1) Quit (Ping timeout: 480 seconds)
[9:36] * analbeard (~shw@support.memset.com) has joined #ceph
[9:37] * overclk (~vshankar@121.244.87.117) Quit (Ping timeout: 480 seconds)
[9:39] * daviddcc (~dcasier@80.12.55.131) has joined #ceph
[9:40] * KaneK (~kane@cpe-172-88-240-14.socal.res.rr.com) Quit (Quit: KaneK)
[9:41] * KaneK (~kane@cpe-172-88-240-14.socal.res.rr.com) has joined #ceph
[9:42] * brian_ (~textual@109.255.114.93) has joined #ceph
[9:46] * allaok (~allaok@machine107.orange-labs.com) has joined #ceph
[9:46] * nils_ (~nils_@doomstreet.collins.kg) has joined #ceph
[9:46] * bitserker (~toni@88.87.194.130) has joined #ceph
[9:49] * brian (~textual@109.255.114.93) Quit (Ping timeout: 480 seconds)
[9:49] * bara (~bara@213.175.37.10) has joined #ceph
[9:49] * wjw-freebsd (~wjw@smtp.digiware.nl) has joined #ceph
[9:50] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[9:50] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[9:52] * bitserker (~toni@88.87.194.130) Quit (Remote host closed the connection)
[9:53] * kefu (~kefu@114.92.107.250) has joined #ceph
[9:55] * danieagle (~Daniel@201-26-27-10.dsl.telesp.net.br) Quit (Quit: Obrigado por Tudo! :-) inte+ :-))
[9:58] * kefu (~kefu@114.92.107.250) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[9:58] * jordanP (~jordan@204.13-14-84.ripe.coltfrance.com) has joined #ceph
[10:01] * pabluk_ is now known as pabluk
[10:02] * TMM (~hp@178-84-46-106.dynamic.upc.nl) Quit (Quit: Ex-Chat)
[10:02] * bjozet (~bjozet@82-183-17-144.customers.ownit.se) Quit (Quit: leaving)
[10:03] * wyang (~wyang@46.21.158.66) Quit (Remote host closed the connection)
[10:04] * bjozet (~bjozet@82-183-17-144.customers.ownit.se) has joined #ceph
[10:05] * theanalyst__ (~abhishekl@223.228.132.52) Quit (Ping timeout: 480 seconds)
[10:05] * thomnico (~thomnico@2a01:e35:8b41:120:2d08:23bb:5f29:f811) Quit (Remote host closed the connection)
[10:09] * garphy`aw is now known as garphy
[10:10] * daviddcc (~dcasier@80.12.55.131) Quit (Ping timeout: 480 seconds)
[10:10] * ram_ (~oftc-webi@static-202-65-140-146.pol.net.in) Quit (Ping timeout: 480 seconds)
[10:11] * fdmanana (~fdmanana@2001:8a0:6dfd:6d01:c985:bab3:55c1:eca7) has joined #ceph
[10:12] * thomnico (~thomnico@2a01:e35:8b41:120:1567:66da:63b9:3677) has joined #ceph
[10:14] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[10:14] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[10:14] * nardial (~ls@dslb-178-006-188-146.178.006.pools.vodafone-ip.de) has joined #ceph
[10:16] * wyang (~wyang@46.21.158.66) has joined #ceph
[10:24] * Neon (~FNugget@tor2r.ins.tor.net.eu.org) has joined #ceph
[10:24] <boolman> I accidentally stopped both my mds server. now my mds cluster is degraded, now both of them is stuck in "up:rejoin", any tip ?
[10:25] <boolman> heartbeat_map is_healthy 'MDSRank' had timed out after 15 \ _send skipping beacon, heartbeat map not healthy
[10:29] * botho (~botho@pD9F92A7F.dip0.t-ipconnect.de) has joined #ceph
[10:32] * derjohn_mob (~aj@p578b6aa1.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[10:37] * thomnico (~thomnico@2a01:e35:8b41:120:1567:66da:63b9:3677) Quit (Quit: Ex-Chat)
[10:37] * thomnico (~thomnico@2a01:e35:8b41:120:1567:66da:63b9:3677) has joined #ceph
[10:37] * Nicola-1980 (~Nicola-19@2-234-77-205.ip222.fastwebnet.it) Quit (Quit: Leaving...)
[10:43] * naoto_ (~naotok@27.131.11.254) Quit (Remote host closed the connection)
[10:45] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[10:45] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[10:47] * mhuang (~mhuang@119.254.120.72) Quit (Quit: Leaving)
[10:49] <boolman> paste: http://ix.io/mIp
[10:52] * rdas (~rdas@106.221.132.163) has joined #ceph
[10:53] * Neon (~FNugget@5P6AABWGE.tor-irc.dnsbl.oftc.net) Quit ()
[10:54] * rakesh_ (~rakesh@106.51.26.185) Quit (Quit: Leaving)
[10:54] * krypto (~krypto@65.115.222.48) Quit (Ping timeout: 480 seconds)
[10:55] * osso (~osso@sgp01-1-78-233-150-179.fbx.proxad.net) has joined #ceph
[10:58] * nardial (~ls@dslb-178-006-188-146.178.006.pools.vodafone-ip.de) Quit (Quit: Leaving)
[10:59] * LeaChim (~LeaChim@host86-185-146-193.range86-185.btcentralplus.com) has joined #ceph
[10:59] * naoto (~naotok@2401:bd00:b001:8920:27:131:11:254) has joined #ceph
[10:59] * ngoswami (~ngoswami@121.244.87.116) has joined #ceph
[11:01] * derjohn_mob (~aj@fw.gkh-setu.de) has joined #ceph
[11:06] * naoto_ (~naotok@27.131.11.254) has joined #ceph
[11:07] * naoto__ (~naotok@2401:bd00:b001:8920:27:131:11:254) has joined #ceph
[11:07] * naoto_ (~naotok@27.131.11.254) Quit (Read error: Connection reset by peer)
[11:13] * naoto (~naotok@2401:bd00:b001:8920:27:131:11:254) Quit (Ping timeout: 480 seconds)
[11:14] * branto1 (~branto@ip-78-102-208-28.net.upcbroadband.cz) has joined #ceph
[11:15] * Wielebny (~Icedove@wr1.contium.pl) Quit (Remote host closed the connection)
[11:15] * derjohn_mob (~aj@fw.gkh-setu.de) Quit (Ping timeout: 480 seconds)
[11:16] * sileht (~sileht@sileht.net) Quit (Quit: WeeChat 1.3)
[11:16] * Wielebny (~Icedove@wr1.contium.pl) has joined #ceph
[11:19] * sileht (~sileht@sileht.net) has joined #ceph
[11:19] * sileht (~sileht@sileht.net) Quit ()
[11:20] * sileht (~sileht@sileht.net) has joined #ceph
[11:22] * mhuang (~mhuang@119.254.120.72) has joined #ceph
[11:22] * kefu (~kefu@114.92.107.250) has joined #ceph
[11:22] * brutuscat (~brutuscat@105.34.133.37.dynamic.jazztel.es) has joined #ceph
[11:25] * derjohn_mob (~aj@fw.gkh-setu.de) has joined #ceph
[11:26] * baotiao (~baotiao@218.30.116.10) has joined #ceph
[11:33] * kefu is now known as kefu|afk
[11:35] * kefu|afk is now known as kefu
[11:38] * naoto (~naotok@27.131.11.254) has joined #ceph
[11:38] * kefu is now known as kefu|afk
[11:40] * Destreyf (~quassel@email.newagecomputers.info) Quit (Read error: Connection reset by peer)
[11:43] <Heebie> Has anyone figured out a good way to expose RBD's via iSCSI to client machines? tgt gave me great performance, but entirely stopped working after a reboot of the iscsi server. (The backing RBD's were still there, but tgt simply would not serve them.) istgt's performance is absolutely abysmal. (1Mbps maximum, where tgt maxed out the iSCSI-facing NIC)
[11:43] * naoto__ (~naotok@2401:bd00:b001:8920:27:131:11:254) Quit (Ping timeout: 480 seconds)
[11:44] * kefu|afk (~kefu@114.92.107.250) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[11:46] * TMM (~hp@sams-office-nat.tomtomgroup.com) has joined #ceph
[11:50] <darkfader> Heebie: i didn't try with ceph in any recent time but generally only use LIO any more
[11:51] <darkfader> it's the only one that has nice masking
[11:51] <darkfader> but i'm not sure if it had a as native rbd backend as tgtd had
[11:51] * linjan (~linjan@176.195.68.96) has joined #ceph
[11:52] <darkfader> but personally i think that tgtd and "the other thing" are like kid's toys while lio is about storage management
[11:52] <Heebie> I didn't find a native for tgt. :/ I found something on native for istgt, but it doesn't appear to work. (It just gives an error) I'll take a look at lio. Thanks.
[11:53] <Heebie> I think "istgt" must stand for "I'm S### TGT"
[11:53] * robaman (~robaman@31.216.37.52) has joined #ceph
[11:55] * TMM (~hp@sams-office-nat.tomtomgroup.com) Quit (Quit: Ex-Chat)
[11:56] * proc_ (~proc@213.180.65.2) has joined #ceph
[11:56] <boolman> I accidentally stopped both my mds server. now my mds cluster is degraded, now both of them is stuck in "up:rejoin", http://ix.io/mIr . any tip on how to get things up and running again?
[11:56] <robaman> anyone know why package ceph-deploy from http://download.ceph.com/debian-infernalis/dists/trusty/main/ doesn't override ceph-deploy in ubuntu 14.04?
[11:58] <robaman> ceph-deploy seems to be missing from http://download.ceph.com/debian-infernalis/dists/trusty/main/binary-amd64/Packages, don't know if thats normal or not
[11:58] * haomaiwang (~haomaiwan@103.15.217.218) Quit (Remote host closed the connection)
[12:00] <Heebie> Wow. Installing "targetcli" (LIO package in Ubuntu) seems to want to pull in an enormous amount of GUI. =O
[12:05] * badone (~badone@66.187.239.16) Quit (Remote host closed the connection)
[12:09] <Heebie> darkfader: It looks like targetcli will also do fibrechannel? (I'd imagine the network transport is the only real difference between the two)
[12:10] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit (Ping timeout: 480 seconds)
[12:10] <darkfader> yes
[12:10] * badone (~badone@66.187.239.16) has joined #ceph
[12:11] <darkfader> i didn't know there was a gui
[12:11] <darkfader> it's just a weird python cli normally
[12:11] <darkfader> it can do fibrechannel target, but last time i tried that i got stuck because the fedora i used was not new enough (was newest that was not in beta)
[12:12] <darkfader> fc20 times, so should be fine now
[12:12] <darkfader> i needed a qlogic hba
[12:12] <darkfader> it can also do infiniband, and even iSER stuff
[12:12] <darkfader> pretty much everthing
[12:12] <darkfader> if you use it heavily, make sure you backup the xml config in /var/lib/targetcli or so
[12:12] <darkfader> it has all the bindings
[12:13] * baotiao (~baotiao@218.30.116.10) Quit (Quit: baotiao)
[12:13] * zhaochao (~zhaochao@60.206.230.18) Quit (Quit: ChatZilla 0.9.92 [Iceweasel 38.4.0/20151103222033])
[12:14] <Heebie> Thanks for the advice. I don't know if it has a GUI, since it's on a machine without a GUI nor access, but the CLI seems pretty darned impressive so far! Do you just use "iblock" type for RBD's?
[12:16] <Mons> hi,
[12:17] <Mons> is there a way to see the state of a remove snapshot operation ?
[12:18] <Mons> I removed a pool snapshot and it take a long time
[12:23] * jayjay (~jjs@dsl-087-195-118-160.solcon.nl) has joined #ceph
[12:24] * EinstCrazy (~EinstCraz@111.30.21.47) Quit (Remote host closed the connection)
[12:24] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[12:24] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[12:28] * jakekosberg (~MKoR@109.201.143.40) has joined #ceph
[12:31] * oms101 (~oms101@p20030057EA02CA00C6D987FFFE4339A1.dip0.t-ipconnect.de) has joined #ceph
[12:32] * bene2 (~bene@2601:18c:8300:f3ae:ea2a:eaff:fe08:3c7a) has joined #ceph
[12:32] * bvi (~bastiaan@185.56.32.1) has joined #ceph
[12:34] * bvi (~bastiaan@185.56.32.1) Quit ()
[12:35] * bene2 (~bene@2601:18c:8300:f3ae:ea2a:eaff:fe08:3c7a) Quit ()
[12:35] * bene2 (~bene@2601:18c:8300:f3ae:ea2a:eaff:fe08:3c7a) has joined #ceph
[12:40] * ade (~abradshaw@dslb-188-103-023-049.188.103.pools.vodafone-ip.de) Quit (Quit: Too sexy for his shirt)
[12:41] * ade (~abradshaw@dslb-188-103-023-049.188.103.pools.vodafone-ip.de) has joined #ceph
[12:43] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[12:43] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[12:51] * EinstCrazy (~EinstCraz@218.69.65.22) has joined #ceph
[12:52] * bitserker (~toni@88.87.194.130) has joined #ceph
[12:55] * naoto (~naotok@27.131.11.254) Quit (Quit: Leaving...)
[12:56] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[12:56] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[12:58] * jakekosberg (~MKoR@7V7AABTAQ.tor-irc.dnsbl.oftc.net) Quit ()
[12:59] <proc_> Hi, I got and issue trying to set up the first monitor for my 3 node + 1 admin server. Issuing ceph-deploy mon create-initial fails with: [node1][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node1.asok mon_status
[12:59] <proc_> [node1][ERROR ] admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
[12:59] * kefu (~kefu@114.92.107.250) has joined #ceph
[12:59] <proc_> There are no errors before that, and the directory /var/run/ceph exists
[13:00] * ade (~abradshaw@dslb-188-103-023-049.188.103.pools.vodafone-ip.de) Quit (Quit: Too sexy for his shirt)
[13:00] <proc_> Using the infernalis release
[13:00] <proc_> This problem didn't appear when I tried firefly
[13:01] <jayjay> check the permissions
[13:01] <jayjay> infernalis runs as ceph
[13:02] * rotbeard (~redbeard@185.32.80.238) has joined #ceph
[13:02] * yanzheng (~zhyan@182.139.205.155) has joined #ceph
[13:05] * i_m (~ivan.miro@deibp9eh1--blueice3n1.emea.ibm.com) has joined #ceph
[13:06] * ade (~abradshaw@dslb-188-103-023-049.188.103.pools.vodafone-ip.de) has joined #ceph
[13:07] <proc_> ah ok
[13:07] <proc_> I will do
[13:08] <proc_> the directory is owned by ceph:ceph
[13:11] * kefu (~kefu@114.92.107.250) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[13:11] * tenshi (~David@office-mtl1-nat-146-218-70-69.gtcomm.net) has joined #ceph
[13:17] * tenshi2 (~David@MTRLPQ42-1176054809.sdsl.bell.ca) Quit (Ping timeout: 480 seconds)
[13:18] * fsimonce (~simon@host28-31-dynamic.30-79-r.retail.telecomitalia.it) Quit (Remote host closed the connection)
[13:18] * fsimonce (~simon@host28-31-dynamic.30-79-r.retail.telecomitalia.it) has joined #ceph
[13:24] * tenshi1 (~David@MTRLPQ42-1176054809.sdsl.bell.ca) has joined #ceph
[13:28] * wjw-freebsd (~wjw@smtp.digiware.nl) Quit (Ping timeout: 480 seconds)
[13:30] * tenshi (~David@office-mtl1-nat-146-218-70-69.gtcomm.net) Quit (Ping timeout: 480 seconds)
[13:32] * TMM (~hp@sams-office-nat.tomtomgroup.com) has joined #ceph
[13:33] * kanagaraj (~kanagaraj@121.244.87.117) Quit (Ping timeout: 480 seconds)
[13:33] <hkraal_> Someone here in charge of the seph.com website or has a contact? It seems that IPv6 is dead in the water for a couple of days in a row now? Route ends on 2607:f298:5:110d:f816:3eff:fe79:ad5d for me
[13:34] * rdas (~rdas@106.221.132.163) Quit (Quit: Leaving)
[13:35] * nardial (~ls@dslb-178-006-188-146.178.006.pools.vodafone-ip.de) has joined #ceph
[13:36] * wjw-freebsd (~wjw@vpn.ecoracks.nl) has joined #ceph
[13:43] * kanagaraj (~kanagaraj@121.244.87.124) has joined #ceph
[13:45] * bene2 (~bene@2601:18c:8300:f3ae:ea2a:eaff:fe08:3c7a) Quit (Quit: Konversation terminated!)
[13:46] <swami1> loicd: HI
[13:50] * kefu (~kefu@114.92.107.250) has joined #ceph
[13:51] * gucki (~smuxi@176.2.133.210) has joined #ceph
[13:52] * fcape (~fcape@107-220-57-73.lightspeed.austtx.sbcglobal.net) has joined #ceph
[13:54] * ira (~ira@c-73-238-173-100.hsd1.ma.comcast.net) has joined #ceph
[13:54] <loicd> swami1: hi
[13:55] * georgem (~Adium@184.151.190.225) has joined #ceph
[13:55] * nardial (~ls@dslb-178-006-188-146.178.006.pools.vodafone-ip.de) Quit (Quit: Leaving)
[13:58] * daviddcc (~dcasier@LAubervilliers-656-1-16-160.w217-128.abo.wanadoo.fr) has joined #ceph
[13:58] * kanagaraj (~kanagaraj@121.244.87.124) Quit (Quit: Leaving)
[14:00] * nhm (~nhm@c-50-171-139-246.hsd1.mn.comcast.net) has joined #ceph
[14:00] * ChanServ sets mode +o nhm
[14:01] * kanagaraj_ (~kanagaraj@121.244.87.117) has joined #ceph
[14:01] * kefu (~kefu@114.92.107.250) Quit (Max SendQ exceeded)
[14:01] * kefu (~kefu@114.92.107.250) has joined #ceph
[14:02] * bara (~bara@213.175.37.10) Quit (Ping timeout: 480 seconds)
[14:05] * kefu is now known as kefu|afk
[14:06] * kefu|afk (~kefu@114.92.107.250) Quit ()
[14:11] * brutuscat (~brutuscat@105.34.133.37.dynamic.jazztel.es) Quit (Remote host closed the connection)
[14:11] * dugravot6 (~dugravot6@dn-infra-04.lionnois.univ-lorraine.fr) Quit (Quit: Leaving.)
[14:12] * jcsp (~jspray@82-71-16-249.dsl.in-addr.zen.co.uk) has joined #ceph
[14:14] * bara (~bara@nat-pool-brq-u.redhat.com) has joined #ceph
[14:14] * swami1 (~swami@49.44.57.245) Quit (Quit: Leaving.)
[14:14] * ade (~abradshaw@dslb-188-103-023-049.188.103.pools.vodafone-ip.de) Quit (Quit: Too sexy for his shirt)
[14:16] * Concubidated (~Adium@pool-98-119-93-148.lsanca.fios.verizon.net) Quit (Quit: Leaving.)
[14:17] * krypto (~krypto@65.115.222.48) has joined #ceph
[14:18] * ade (~abradshaw@dslb-188-103-023-049.188.103.pools.vodafone-ip.de) has joined #ceph
[14:22] * krypto (~krypto@65.115.222.48) Quit (Quit: Leaving)
[14:22] * krypto (~krypto@65.115.222.48) has joined #ceph
[14:23] * karnan (~karnan@121.244.87.117) Quit (Remote host closed the connection)
[14:30] * georgem (~Adium@184.151.190.225) Quit (Quit: Leaving.)
[14:31] * dugravot6 (~dugravot6@129.131.135.77.rev.sfr.net) has joined #ceph
[14:33] * dugravot61 (~dugravot6@178.251.23.21) has joined #ceph
[14:36] * Concubidated (~Adium@pool-98-119-93-148.lsanca.fios.verizon.net) has joined #ceph
[14:36] * dugravot61 (~dugravot6@178.251.23.21) Quit ()
[14:39] * dugravot6 (~dugravot6@129.131.135.77.rev.sfr.net) Quit (Ping timeout: 480 seconds)
[14:47] * fedgoat (~theengine@45-31-177-36.lightspeed.austtx.sbcglobal.net) has joined #ceph
[14:47] * kanagaraj_ (~kanagaraj@121.244.87.117) Quit (Quit: Leaving)
[14:47] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[14:47] * vikhyat (~vumrao@121.244.87.116) Quit (Ping timeout: 480 seconds)
[14:47] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[14:48] * owasserm (~owasserm@a212-238-239-152.adsl.xs4all.nl) Quit (Quit: Ex-Chat)
[14:48] * ircolle (~ircolle@nat-pool-rdu-u.redhat.com) has joined #ceph
[14:50] * shaunm (~shaunm@208.102.161.229) has joined #ceph
[14:50] * ircolle (~ircolle@nat-pool-rdu-u.redhat.com) has left #ceph
[14:53] * georgem (~Adium@206.108.127.16) has joined #ceph
[14:54] * jdillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) has joined #ceph
[14:55] * allaok (~allaok@machine107.orange-labs.com) has left #ceph
[14:55] * pabluk is now known as pabluk_
[14:57] * vikhyat (~vumrao@114.143.75.111) has joined #ceph
[14:59] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[14:59] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[15:00] * krypto (~krypto@65.115.222.48) Quit (Quit: Leaving)
[15:00] * owasserm (~owasserm@2001:984:d3f7:1:5ec5:d4ff:fee0:f6dc) has joined #ceph
[15:00] * owasserm (~owasserm@2001:984:d3f7:1:5ec5:d4ff:fee0:f6dc) Quit ()
[15:00] * wulb (~wulb@123.58.191.68) Quit (Ping timeout: 480 seconds)
[15:00] * wulb (~wulb@123.58.191.68) has joined #ceph
[15:01] * logan (~a@216.144.251.246) has joined #ceph
[15:01] * jasuarez (~jasuarez@38.Red-81-38-77.dynamicIP.rima-tde.net) Quit (Ping timeout: 480 seconds)
[15:04] * owasserm (~owasserm@2001:984:d3f7:1:5ec5:d4ff:fee0:f6dc) has joined #ceph
[15:05] * owasserm (~owasserm@2001:984:d3f7:1:5ec5:d4ff:fee0:f6dc) Quit ()
[15:05] * owasserm (~owasserm@2001:984:d3f7:1:5ec5:d4ff:fee0:f6dc) has joined #ceph
[15:06] * pabluk_ is now known as pabluk
[15:07] * fcape (~fcape@107-220-57-73.lightspeed.austtx.sbcglobal.net) Quit (Ping timeout: 480 seconds)
[15:09] * haomaiwang (~haomaiwan@103.15.217.218) has joined #ceph
[15:10] * Exchizz (~kvirc@stud-128.sdu.dk) has joined #ceph
[15:14] * mhack (~mhack@nat-pool-bos-t.redhat.com) has joined #ceph
[15:14] * bene2 (~bene@nat-pool-bos-t.redhat.com) has joined #ceph
[15:16] * Dave (~oftc-webi@192.43.161.1) has joined #ceph
[15:16] * Dave is now known as Guest781
[15:17] * johnavp1989 (~jpetrini@8.39.115.8) has joined #ceph
[15:17] * kefu (~kefu@114.92.107.250) has joined #ceph
[15:17] <Guest781> I have a ceph-osd stuck in a "booting" state, and can't figure out why. Anyone have thoughts?
[15:19] * nardial (~ls@dslb-178-006-188-146.178.006.pools.vodafone-ip.de) has joined #ceph
[15:19] * gucki (~smuxi@176.2.133.210) Quit (Ping timeout: 480 seconds)
[15:21] * brad_mssw (~brad@66.129.88.50) has joined #ceph
[15:26] * yanzheng (~zhyan@182.139.205.155) Quit (Ping timeout: 480 seconds)
[15:27] * nhm (~nhm@c-50-171-139-246.hsd1.mn.comcast.net) Quit (Ping timeout: 480 seconds)
[15:27] * AG_Scott (~Scrin@185.101.107.136) has joined #ceph
[15:31] * geert_ (~geert@87.213.229.226) has joined #ceph
[15:32] * KHJared (~KHJared@65.99.235.248) Quit (Remote host closed the connection)
[15:34] * dyasny (~dyasny@104.158.24.36) Quit (Ping timeout: 480 seconds)
[15:35] <geert_> hello. Anyone else getting 404 errors from the gitbuilder.ceph.com yum repo?
[15:35] * yanzheng (~zhyan@182.139.205.155) has joined #ceph
[15:35] * fcape (~fcape@rrcs-97-77-228-30.sw.biz.rr.com) has joined #ceph
[15:36] <geert_> http://gitbuilder.ceph.com/mod_fastcgi-rpm-rhel7-x86_64-basic/ref/master/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found
[15:36] * rwheeler (~rwheeler@pool-173-48-214-9.bstnma.fios.verizon.net) Quit (Quit: Leaving)
[15:39] * mattbenjamin (~mbenjamin@76-206-42-105.lightspeed.livnmi.sbcglobal.net) has joined #ceph
[15:40] * yanzheng (~zhyan@182.139.205.155) Quit ()
[15:43] * brutuscat (~brutuscat@188.Red-81-43-172.staticIP.rima-tde.net) has joined #ceph
[15:45] * lpabon (~quassel@24-151-54-34.dhcp.nwtn.ct.charter.com) has joined #ceph
[15:45] * dyasny (~dyasny@104.158.24.36) has joined #ceph
[15:49] * kefu is now known as kefu|afk
[15:50] * harold (~hamiller@71-94-227-123.dhcp.mdfd.or.charter.com) has joined #ceph
[15:50] * kefu|afk (~kefu@114.92.107.250) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[15:53] * nardial (~ls@dslb-178-006-188-146.178.006.pools.vodafone-ip.de) Quit (Quit: Leaving)
[15:54] * gucki (~smuxi@176.2.133.210) has joined #ceph
[15:55] * Exchizz (~kvirc@stud-128.sdu.dk) Quit (Ping timeout: 480 seconds)
[15:57] * AG_Scott (~Scrin@7V7AABTDH.tor-irc.dnsbl.oftc.net) Quit ()
[15:57] * nhm (~nhm@c-50-171-139-246.hsd1.mn.comcast.net) has joined #ceph
[15:57] * ChanServ sets mode +o nhm
[15:57] * kefu (~kefu@114.92.107.250) has joined #ceph
[15:57] * harold (~hamiller@71-94-227-123.dhcp.mdfd.or.charter.com) Quit (Quit: Leaving)
[16:00] * i_m (~ivan.miro@deibp9eh1--blueice3n1.emea.ibm.com) Quit (Ping timeout: 480 seconds)
[16:00] * xarses_ (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[16:01] * i_m (~ivan.miro@deibp9eh1--blueice3n3.emea.ibm.com) has joined #ceph
[16:02] * enax (~enax@hq.ezit.hu) Quit (Ping timeout: 480 seconds)
[16:03] * Guest781 (~oftc-webi@192.43.161.1) Quit (Quit: Page closed)
[16:03] * dlan_ (~dennis@116.228.88.131) has joined #ceph
[16:04] * bara (~bara@nat-pool-brq-u.redhat.com) Quit (Ping timeout: 480 seconds)
[16:05] * dlan (~dennis@116.228.88.131) Quit (Read error: Connection reset by peer)
[16:05] * zaitcev (~zaitcev@c-50-130-189-82.hsd1.nm.comcast.net) has joined #ceph
[16:08] <DeMiNe0_> Hi guys. I'm running a ceph cluster here for testing purposes. We're going to push it into production pretty soon, and I'm looking for the best way to test ceph osd recovery while at the same time swapping some smaller 500gb OSD drives for larger 2tb drives.
[16:08] * cholcombe (~chris@c-73-180-29-35.hsd1.or.comcast.net) has joined #ceph
[16:09] <DeMiNe0_> My idea was just to start pulling out drives, marking the OSD as lost, then replacing it after the rebalance with a 2tb drive. I'm not sure how to see what OSD's data exists on though. It would help to know this information to test the recovery portion of this test.
[16:10] * T1w (~jens@node3.survey-it.dk) Quit (Ping timeout: 480 seconds)
[16:10] <DeMiNe0_> rather, I'm not sure where to find what OSD's a certain set (an rbd image for example) exists on.
[16:11] * gregmark (~Adium@68.87.42.115) has joined #ceph
[16:11] * tsg (~tgohad@jfdmzpr01-ext.jf.intel.com) has joined #ceph
[16:12] * tsg (~tgohad@jfdmzpr01-ext.jf.intel.com) Quit ()
[16:12] * xarses_ (~xarses@64.124.158.100) has joined #ceph
[16:13] * jcsp (~jspray@82-71-16-249.dsl.in-addr.zen.co.uk) Quit (Ping timeout: 480 seconds)
[16:16] * smerz (~ircircirc@37.74.194.90) Quit (Remote host closed the connection)
[16:18] * bara (~bara@nat-pool-brq-t.redhat.com) has joined #ceph
[16:18] * dugravot6 (~dugravot6@dn-infra-04.lionnois.univ-lorraine.fr) has joined #ceph
[16:18] * dugravot6 (~dugravot6@dn-infra-04.lionnois.univ-lorraine.fr) Quit ()
[16:22] * kefu is now known as kefu|afk
[16:22] * proc_ (~proc@213.180.65.2) Quit (Quit: leaving)
[16:22] * georgem (~Adium@206.108.127.16) Quit (Ping timeout: 480 seconds)
[16:24] <Heebie> DeMiNe0_: What OSD's the data is contained on would be in the CRUSH map, although you'd need to know how to read the CRUSH map. (which rule/policy applies to the appropriate pool, and what OSD's are contained within the rule/policy)
[16:24] <Heebie> One of the things about ceph is that you shouldn't need to know that data. As soon as you "lose" an OSD, ceph will completely rewrite that map and shift the data around.
[16:24] * dgurtner (~dgurtner@178.197.231.21) Quit (Ping timeout: 480 seconds)
[16:25] * jcsp (~jspray@82-71-16-249.dsl.in-addr.zen.co.uk) has joined #ceph
[16:25] <Heebie> You can get a list somewhere of the PG assignments, but that won't tell you specifics, since the data is spread across all the participating OSD's related to that pool
[16:25] <DeMiNe0_> I understand that, I want to pull an OSD that I know has data on it, so I can check the integrity after the rebuild is done. I think I recall seeing information on how to read the crush map. I'll look into that.
[16:27] <DeMiNe0_> I think as long as I pick an OSD that has PG's with data stored in them assgined to it, the test should work fine
[16:27] <Heebie> Create a pool, give it a crush map that limits it to just a few OSD's. Copy a specific set of information to an RBD in that set. Pull the OSD, and re-check the data. That'll test a sub-set of data that is limited to specific OSD's.
[16:28] <DeMiNe0_> I'll do that, thanks!
[16:29] * kefu|afk (~kefu@114.92.107.250) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[16:30] <Heebie> You can log the output of `ceph status; ceph osd perf ; ceph osd df` to a file and analyse it later. It'll show how many PG's need to be adjusted, progress on adjusting them, performance of OSD's during the data restructuring, and disk usage on the OSD's
[16:30] * rwheeler (~rwheeler@nat-pool-bos-t.redhat.com) has joined #ceph
[16:30] <Heebie> That should help you "visualise" where the data goes, anyhow.
[16:30] <DeMiNe0_> Ahhh, nice. That's really helpful
[16:31] <Heebie> I'm quite new to this myself, so someone else might chime in with something more useful.
[16:32] * georgem (~Adium@206.108.127.16) has joined #ceph
[16:35] <DeMiNe0_> It's been a pretty awesome little experience learning this stuff. Every day I seem to be learning something new about ceph. I'm about to embark on the OpenStack+ceph journey this week, heh
[16:40] * moore (~moore@71-211-73-118.phnx.qwest.net) has joined #ceph
[16:41] * moore (~moore@71-211-73-118.phnx.qwest.net) Quit (Remote host closed the connection)
[16:41] * moore (~moore@ip-216-69-191-1.ip.secureserver.net) has joined #ceph
[16:41] * herrsergio (~herrsergi@200.77.224.239) has joined #ceph
[16:42] * ngoswami (~ngoswami@121.244.87.116) Quit (Quit: Leaving)
[16:42] * herrsergio is now known as Guest791
[16:43] * madkiss (~madkiss@2001:6f8:12c3:f00f:8815:aa96:c2a3:316a) has joined #ceph
[16:48] * tsg (~tgohad@jfdmzpr01-ext.jf.intel.com) has joined #ceph
[16:50] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[16:50] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[16:56] * analbeard (~shw@support.memset.com) Quit (Quit: Leaving.)
[16:57] * jclm (~jclm@ip68-108-16-17.lv.lv.cox.net) has joined #ceph
[16:57] * jclm (~jclm@ip68-108-16-17.lv.lv.cox.net) Quit ()
[16:57] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[16:57] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[16:57] * haomaiwang (~haomaiwan@103.15.217.218) Quit (Remote host closed the connection)
[16:57] * fcape (~fcape@rrcs-97-77-228-30.sw.biz.rr.com) Quit (Ping timeout: 480 seconds)
[16:58] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) has joined #ceph
[16:59] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) Quit (Remote host closed the connection)
[16:59] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) has joined #ceph
[17:00] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) Quit (Remote host closed the connection)
[17:00] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) has joined #ceph
[17:01] * haomaiwang (~haomaiwan@60-250-10-240.HINET-IP.hinet.net) Quit (Remote host closed the connection)
[17:01] * haomaiwang (~haomaiwan@103.15.217.218) has joined #ceph
[17:02] * fcape (~fcape@38.67.18.51) has joined #ceph
[17:06] * linjan_ (~linjan@176.193.207.13) has joined #ceph
[17:09] * Guest791 (~herrsergi@200.77.224.239) Quit (Ping timeout: 480 seconds)
[17:10] * Exchizz (~kvirc@stud-128.sdu.dk) has joined #ceph
[17:12] * fcape_ (~fcape@rrcs-97-77-228-30.sw.biz.rr.com) has joined #ceph
[17:12] * linjan (~linjan@176.195.68.96) Quit (Ping timeout: 480 seconds)
[17:14] * fcape (~fcape@38.67.18.51) Quit (Ping timeout: 480 seconds)
[17:20] * nardial (~ls@dslb-178-006-188-146.178.006.pools.vodafone-ip.de) has joined #ceph
[17:22] * yuriw1 (~Adium@2601:645:4380:112c:52a:559f:d49a:9baf) Quit (Quit: Leaving.)
[17:23] * reed (~reed@75-101-54-18.dsl.static.fusionbroadband.com) has joined #ceph
[17:23] * yuriw (~Adium@2601:645:4380:112c:52a:559f:d49a:9baf) has joined #ceph
[17:23] * TMM (~hp@sams-office-nat.tomtomgroup.com) Quit (Quit: Ex-Chat)
[17:24] * b0e (~aledermue@213.95.25.82) Quit (Quit: Leaving.)
[17:30] * KaneK (~kane@cpe-172-88-240-14.socal.res.rr.com) Quit (Quit: KaneK)
[17:32] * Exchizz (~kvirc@stud-128.sdu.dk) Quit (Ping timeout: 480 seconds)
[17:36] * overclk (~vshankar@59.93.65.224) has joined #ceph
[17:36] * i_m (~ivan.miro@deibp9eh1--blueice3n3.emea.ibm.com) Quit (Ping timeout: 480 seconds)
[17:38] * sudocat (~dibarra@192.185.1.20) has joined #ceph
[17:40] * toMeloos (~toMeloos@53568B3D.cm-6-7c.dynamic.ziggo.nl) Quit (Ping timeout: 480 seconds)
[17:40] * haomaiwang (~haomaiwan@103.15.217.218) Quit (Remote host closed the connection)
[17:40] * yurasfromch (~yuriy@109.122.60.143) has joined #ceph
[17:40] * tsugano (~tsugano@pw126152088103.10.panda-world.ne.jp) has joined #ceph
[17:40] * haomaiwang (~haomaiwan@li715-221.members.linode.com) has joined #ceph
[17:43] * jayjay (~jjs@dsl-087-195-118-160.solcon.nl) Quit (Ping timeout: 480 seconds)
[17:46] * lpabon (~quassel@24-151-54-34.dhcp.nwtn.ct.charter.com) Quit (Remote host closed the connection)
[17:47] * tsugano (~tsugano@pw126152088103.10.panda-world.ne.jp) Quit (Quit: Colloquy for iPhone - http://colloquy.mobi)
[17:49] * Exchizz (~kvirc@wifi-out.sdu.dk) has joined #ceph
[17:51] * wjw-freebsd (~wjw@vpn.ecoracks.nl) Quit (Ping timeout: 480 seconds)
[17:54] * bara (~bara@nat-pool-brq-t.redhat.com) Quit (Ping timeout: 480 seconds)
[17:55] * i_m (~ivan.miro@deibp9eh1--blueice1n0.emea.ibm.com) has joined #ceph
[18:00] * nardial (~ls@dslb-178-006-188-146.178.006.pools.vodafone-ip.de) Quit (Quit: Leaving)
[18:01] * haomaiwang (~haomaiwan@li715-221.members.linode.com) Quit (Remote host closed the connection)
[18:01] * gucki_ (~smuxi@176.2.133.210) has joined #ceph
[18:01] * gucki_ (~smuxi@176.2.133.210) Quit (Remote host closed the connection)
[18:01] * haomaiwang (~haomaiwan@li715-221.members.linode.com) has joined #ceph
[18:01] * gucki (~smuxi@176.2.133.210) Quit (Remote host closed the connection)
[18:01] * gucki (~smuxi@176.2.133.210) has joined #ceph
[18:03] * KaneK (~kane@12.206.204.58) has joined #ceph
[18:05] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[18:05] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[18:06] * vikhyat (~vumrao@114.143.75.111) Quit (Quit: Leaving)
[18:07] * mattbenjamin (~mbenjamin@76-206-42-105.lightspeed.livnmi.sbcglobal.net) Quit (Quit: Leaving.)
[18:11] * KaneK (~kane@12.206.204.58) Quit (Quit: KaneK)
[18:11] * botho (~botho@pD9F92A7F.dip0.t-ipconnect.de) Quit (Quit: Verlassend)
[18:13] * swami1 (~swami@27.7.160.191) has joined #ceph
[18:15] * branto1 (~branto@ip-78-102-208-28.net.upcbroadband.cz) Quit (Quit: Leaving.)
[18:17] * overclk (~vshankar@59.93.65.224) Quit (Quit: Zzzz...)
[18:18] * i_m (~ivan.miro@deibp9eh1--blueice1n0.emea.ibm.com) Quit (Ping timeout: 480 seconds)
[18:19] * geert_ (~geert@87.213.229.226) Quit (Remote host closed the connection)
[18:22] * jwilkins (~jowilkin@2601:644:4000:97c0::4a04) has joined #ceph
[18:26] * LPG (~LPG@c-50-181-212-148.hsd1.wa.comcast.net) has joined #ceph
[18:26] <Heebie> I'm seeing significantly slow iscsi performance using lio/targetcli + rbd for the backing store. Are there some settings to tweak? (The same rbd's performed significantly faster under "tgt" until they stopped working.)
[18:27] * TMM (~hp@178-84-46-106.dynamic.upc.nl) has joined #ceph
[18:29] * jordanP (~jordan@204.13-14-84.ripe.coltfrance.com) Quit (Quit: Leaving)
[18:29] * vasu (~vasu@c-73-231-60-138.hsd1.ca.comcast.net) has joined #ceph
[18:31] * davidzlap (~Adium@cpe-172-91-154-245.socal.res.rr.com) has joined #ceph
[18:32] * LPG_ (~LPG@c-50-181-212-148.hsd1.wa.comcast.net) Quit (Ping timeout: 480 seconds)
[18:33] * yurasfromch (~yuriy@109.122.60.143) Quit ()
[18:33] * pabluk is now known as pabluk_
[18:35] * egonzalez (~egonzalez@238.238.14.62.static.jazztel.es) Quit (Quit: Saliendo)
[18:39] * davidzlap1 (~Adium@2605:e000:1313:8003:9576:217c:26b8:8622) Quit (Ping timeout: 480 seconds)
[18:39] * LPG (~LPG@c-50-181-212-148.hsd1.wa.comcast.net) Quit (Ping timeout: 480 seconds)
[18:40] * krypto (~krypto@G68-121-13-130.sbcis.sbc.com) has joined #ceph
[18:41] * mattbenjamin (~mbenjamin@aa2.linuxbox.com) has joined #ceph
[18:41] <Heebie> Using RAM as the backing store for the same clients seems only limited by the speed of the network in-between, so it appears to be something in between ceph and iscsi.
[18:44] * johnavp1989 (~jpetrini@8.39.115.8) Quit (Ping timeout: 480 seconds)
[18:46] * mhuang (~mhuang@119.254.120.72) Quit (Quit: This computer has gone to sleep)
[18:50] * garphy is now known as garphy`aw
[18:57] * moore_ (~moore@71-211-73-118.phnx.qwest.net) has joined #ceph
[18:58] * johnavp1989 (~jpetrini@8.39.115.8) has joined #ceph
[18:58] * TheSov2 (~TheSov@cip-248.trustwave.com) has joined #ceph
[19:01] * haomaiwang (~haomaiwan@li715-221.members.linode.com) Quit (Remote host closed the connection)
[19:01] * haomaiwang (~haomaiwan@li715-221.members.linode.com) has joined #ceph
[19:04] * LPG (~LPG@c-50-181-212-148.hsd1.wa.comcast.net) has joined #ceph
[19:04] * moore (~moore@ip-216-69-191-1.ip.secureserver.net) Quit (Ping timeout: 480 seconds)
[19:05] * krypto (~krypto@G68-121-13-130.sbcis.sbc.com) Quit (Ping timeout: 480 seconds)
[19:05] * ibravo (~ibravo@72.198.142.104) has joined #ceph
[19:05] * krypto (~krypto@198.24.6.220) has joined #ceph
[19:16] * lcurtis_ (~lcurtis@47.19.105.250) has joined #ceph
[19:20] * krypto (~krypto@198.24.6.220) Quit (Ping timeout: 480 seconds)
[19:20] * krypto (~krypto@G68-121-13-167.sbcis.sbc.com) has joined #ceph
[19:24] * daviddcc (~dcasier@LAubervilliers-656-1-16-160.w217-128.abo.wanadoo.fr) Quit (Ping timeout: 480 seconds)
[19:24] * zaitcev (~zaitcev@c-50-130-189-82.hsd1.nm.comcast.net) Quit (Quit: Bye)
[19:24] * enax (~enax@94-21-125-92.pool.digikabel.hu) has joined #ceph
[19:28] * swami1 (~swami@27.7.160.191) Quit (Quit: Leaving.)
[19:28] * yguang11 (~yguang11@nat-dip33-wl-g.cfw-a-gci.corp.yahoo.com) has joined #ceph
[19:29] * gregmark (~Adium@68.87.42.115) Quit (Quit: Leaving.)
[19:33] * gregmark (~Adium@68.87.42.115) has joined #ceph
[19:33] * nils_ (~nils_@doomstreet.collins.kg) Quit (Quit: This computer has gone to sleep)
[19:37] * koma (~koma@0001c112.user.oftc.net) has joined #ceph
[19:39] * enax (~enax@94-21-125-92.pool.digikabel.hu) Quit (Ping timeout: 480 seconds)
[19:39] * brutuscat (~brutuscat@188.Red-81-43-172.staticIP.rima-tde.net) Quit (Remote host closed the connection)
[19:39] * gucki (~smuxi@176.2.133.210) Quit (Read error: Connection reset by peer)
[19:40] * KaneK (~kane@12.206.204.58) has joined #ceph
[19:40] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[19:41] * tomc (~tomc@192.41.52.12) has joined #ceph
[19:43] <tomc> Hey folks, I???m trying to get logs uploaded in association with issue 13990, I was told to use ceph-post-file but that returns a connection timeout. Is there any way to get a 250MB+ log file up to the bug?
[19:43] <tomc> whats the recommended way to submit large logs at this point?
[19:46] <gregsfortytwo> tomc: the lab which ceph-post-file targets is being physically relocated
[19:46] <gregsfortytwo> it'll work again in a few days, but we don't provide any good alternatives for large uploads unfortunately
[19:47] <gregsfortytwo> some users don't mind putting it in dropbox (or whatever) and posting a link, is the only other idea I've got besides waiting a bit :/
[19:47] * mykola (~Mikolaj@91.225.201.130) has joined #ceph
[19:47] <tomc> ok, well I can wait a bit.
[19:57] * LPG (~LPG@c-50-181-212-148.hsd1.wa.comcast.net) Quit (Remote host closed the connection)
[19:58] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[19:58] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Remote host closed the connection)
[19:58] * tsg (~tgohad@jfdmzpr01-ext.jf.intel.com) Quit (Remote host closed the connection)
[20:01] * haomaiwang (~haomaiwan@li715-221.members.linode.com) Quit (Remote host closed the connection)
[20:01] * vbellur (~vijay@c-71-234-227-202.hsd1.ma.comcast.net) Quit (Quit: Leaving.)
[20:01] * wulb (~wulb@123.58.191.68) Quit (Ping timeout: 480 seconds)
[20:01] * haomaiwang (~haomaiwan@li715-221.members.linode.com) has joined #ceph
[20:01] * vbellur (~vijay@c-71-234-227-202.hsd1.ma.comcast.net) has joined #ceph
[20:01] * wulb (~wulb@123.58.191.68) has joined #ceph
[20:03] * daviddcc (~dcasier@80.12.55.163) has joined #ceph
[20:04] * brutuscat (~brutuscat@105.34.133.37.dynamic.jazztel.es) has joined #ceph
[20:04] * rikai (~Guest1390@77.247.178.200) has joined #ceph
[20:06] <[arx]> leseb_away: ping
[20:07] * rotbeard (~redbeard@185.32.80.238) Quit (Quit: Leaving)
[20:09] * rwheeler (~rwheeler@nat-pool-bos-t.redhat.com) Quit (Quit: Leaving)
[20:09] * vbellur (~vijay@c-71-234-227-202.hsd1.ma.comcast.net) Quit (Ping timeout: 480 seconds)
[20:10] * johnavp1989 (~jpetrini@8.39.115.8) Quit (Read error: Connection reset by peer)
[20:10] * johnavp1989 (~jpetrini@8.39.115.8) has joined #ceph
[20:16] * johnavp19891 (~jpetrini@8.39.115.8) has joined #ceph
[20:18] * ade (~abradshaw@dslb-188-103-023-049.188.103.pools.vodafone-ip.de) Quit (Ping timeout: 480 seconds)
[20:19] * fcape (~fcape@rrcs-97-77-228-30.sw.biz.rr.com) has joined #ceph
[20:20] * fcape_ (~fcape@rrcs-97-77-228-30.sw.biz.rr.com) Quit (Remote host closed the connection)
[20:21] * johnavp1989 (~jpetrini@8.39.115.8) Quit (Ping timeout: 480 seconds)
[20:25] * daviddcc (~dcasier@80.12.55.163) Quit (Ping timeout: 480 seconds)
[20:26] * ade (~abradshaw@dslb-188-103-023-049.188.103.pools.vodafone-ip.de) has joined #ceph
[20:26] * cathode (~cathode@50-198-166-81-static.hfc.comcastbusiness.net) Quit (Quit: Leaving)
[20:26] * ade (~abradshaw@dslb-188-103-023-049.188.103.pools.vodafone-ip.de) Quit ()
[20:27] * moore_ (~moore@71-211-73-118.phnx.qwest.net) Quit (Remote host closed the connection)
[20:27] * moore (~moore@64.202.160.233) has joined #ceph
[20:28] * brad[] (~Brad@184-94-17-26.dedicated.allstream.net) Quit (Quit: Leaving)
[20:34] * rikai (~Guest1390@5P6AABWTH.tor-irc.dnsbl.oftc.net) Quit ()
[20:37] * jayjay (~jjs@dsl-213-134-254-157.solcon.nl) has joined #ceph
[20:37] <TheSov2> pong
[20:49] * daviddcc (~dcasier@84.197.151.77.rev.sfr.net) has joined #ceph
[20:50] * jayjay (~jjs@dsl-213-134-254-157.solcon.nl) Quit (Ping timeout: 480 seconds)
[20:54] * proc_ (~proc@h-95-179.a463.priv.bahnhof.se) has joined #ceph
[20:55] <proc_> Hi, I asked earlier today, but I really can't figure it out. I've followed the guide on ceph homepage and it seems to work with firefly release. But with infernalis I get these errors:
[20:55] <proc_> [node1][INFO ] Running command: sudo initctl emit ceph-mon cluster=ceph id=node1
[20:55] <proc_> Any ideas?
[20:55] <proc_> [node1][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node1.asok mon_status
[20:55] <proc_> [node1][ERROR ] admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
[20:55] <proc_> [node1][WARNIN] monitor: mon.node1, might not be running yet
[20:56] <proc_> (permissions are correct, logs are empty, ceph:ceph owns /var/run/ceph
[20:57] * Cybertinus (~Cybertinu@cybertinus.customer.cloud.nl) Quit (Ping timeout: 480 seconds)
[21:01] * haomaiwang (~haomaiwan@li715-221.members.linode.com) Quit (Remote host closed the connection)
[21:01] * haomaiwang (~haomaiwan@li715-221.members.linode.com) has joined #ceph
[21:02] * derjohn_mob (~aj@fw.gkh-setu.de) Quit (Ping timeout: 480 seconds)
[21:04] * stiopa (~stiopa@cpc73828-dals21-2-0-cust630.20-2.cable.virginm.net) has joined #ceph
[21:09] * Exchizz (~kvirc@wifi-out.sdu.dk) Quit (Ping timeout: 480 seconds)
[21:14] * tomc (~tomc@192.41.52.12) Quit (Quit: tomc)
[21:14] * daviddcc (~dcasier@84.197.151.77.rev.sfr.net) Quit (Ping timeout: 480 seconds)
[21:15] * brutuscat (~brutuscat@105.34.133.37.dynamic.jazztel.es) Quit (Remote host closed the connection)
[21:15] * Cybertinus (~Cybertinu@cybertinus.customer.cloud.nl) has joined #ceph
[21:15] * brutuscat (~brutuscat@105.34.133.37.dynamic.jazztel.es) has joined #ceph
[21:18] * brutuscat (~brutuscat@105.34.133.37.dynamic.jazztel.es) Quit (Remote host closed the connection)
[21:18] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit (Ping timeout: 480 seconds)
[21:22] * davidzlap1 (~Adium@cpe-172-91-154-245.socal.res.rr.com) has joined #ceph
[21:24] * daviddcc (~dcasier@80.12.59.12) has joined #ceph
[21:27] <alfredodeza> proc_: would you mind pasting the full output of the ceph-deploy command in a paste site so I can take a look?
[21:28] <TheSov2> proc_, what os are u installing on?
[21:28] <alfredodeza> TheSov2: that would be answered with the full paste of the ceph-deploy output :)
[21:29] * davidzlap (~Adium@cpe-172-91-154-245.socal.res.rr.com) Quit (Ping timeout: 480 seconds)
[21:29] <TheSov2> well i was going to link him to a full hammer install which is the same as infernalis but its for debian
[21:29] <TheSov2> and its step by step from scratch
[21:31] * osso (~osso@sgp01-1-78-233-150-179.fbx.proxad.net) Quit (Quit: osso)
[21:35] * cathode (~cathode@50-198-166-81-static.hfc.comcastbusiness.net) has joined #ceph
[21:40] * thomnico (~thomnico@2a01:e35:8b41:120:1567:66da:63b9:3677) Quit (Quit: Ex-Chat)
[21:46] * davidzlap1 (~Adium@cpe-172-91-154-245.socal.res.rr.com) Quit (Ping timeout: 480 seconds)
[21:54] <proc_> TheSov2: that would be good
[21:54] <proc_> alfredodeza: sure, just a sec
[21:54] <proc_> And I'm using Ubuntu 14.04
[21:54] * davidzlap (~Adium@2605:e000:1313:8003:7d53:ed75:bd58:a542) has joined #ceph
[21:55] <proc_> alfredodeza: http://pastebin.com/VUEBmNXV
[21:55] * Maariu5_ (~AluAlu@94.242.195.186) has joined #ceph
[21:56] * garphy`aw is now known as garphy
[21:58] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[21:58] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[21:58] <alfredodeza> proc_: are you trying to create a cluster with just one monitor? that will not work
[21:58] <alfredodeza> you need a quorum and iirc you need at least 2?
[21:59] <alfredodeza> are you adding a monitor to an existing cluster or are you starting from scratch
[22:00] <joshd> it'll work with 1 mon, for a test setup. you need at least 3 to be able to handle one failing
[22:01] * haomaiwang (~haomaiwan@li715-221.members.linode.com) Quit (Remote host closed the connection)
[22:01] <alfredodeza> the output shows that *maybe* this wasn't done from scratch though
[22:01] * haomaiwang (~haomaiwan@li715-221.members.linode.com) has joined #ceph
[22:02] * TheSov3 (~TheSov@204.13.200.248) has joined #ceph
[22:03] * vata1 (~vata@207.96.182.162) has joined #ceph
[22:04] * KaneK (~kane@12.206.204.58) Quit (Quit: KaneK)
[22:06] * baotiao (~baotiao@114.253.119.113) has joined #ceph
[22:07] <proc_> I'm following the guide on ceph webpage
[22:08] <proc_> and first one mon and then I think the guide describes how to add another
[22:08] <proc_> etc
[22:09] * TheSov2 (~TheSov@cip-248.trustwave.com) Quit (Ping timeout: 480 seconds)
[22:09] <alfredodeza> proc_: but in this particular case are you adding one or you are just creating one?
[22:09] <proc_> ceph-deploy mon create-initial (so first one)
[22:10] <alfredodeza> what do logs on that server say?
[22:10] <alfredodeza> tail -f /var/log/ceph/*
[22:11] <proc_> it's empty
[22:11] <proc_> ceph-mon.node1.log
[22:11] <alfredodeza> geez
[22:12] <proc_> stack@node1:~$ ceph -s
[22:12] <proc_> 2015-12-09 22:11:56.997691 7faf8b359700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory
[22:12] <proc_> 2015-12-09 22:11:56.997903 7faf8b359700 -1 monclient(hunting): ERROR: missing keyring, cannot use cephx for authentication
[22:12] <proc_> 2015-12-09 22:11:56.998037 7faf8b359700 0 librados: client.admin initialization error (2) No such file or directory
[22:12] <proc_> Error connecting to cluster: ObjectNotFound
[22:12] <proc_> (sorry for paste)
[22:12] <proc_> but could that be something?
[22:12] <proc_> But as I understood, the keyrings will be created once the monitor is up
[22:12] <proc_> ?
[22:12] <alfredodeza> it is supposed to create the keyring
[22:12] <alfredodeza> right
[22:13] <proc_> ok
[22:13] <alfredodeza> hrmn
[22:13] <alfredodeza> what is going on with your firewall?
[22:13] <alfredodeza> is it on? disabled?
[22:13] <proc_> there are no firewall between the hosts. It's and internal network
[22:13] <proc_> and no rules for iptables
[22:13] * linjan_ (~linjan@176.193.207.13) Quit (Ping timeout: 480 seconds)
[22:14] <alfredodeza> I am running out of ideas :(
[22:14] <proc_> however, there is a firewall in front of them for the public network, but I guess it doesn't matter since its all on internal net
[22:14] * hemebond (~james@121.98.154.114) has joined #ceph
[22:14] <proc_> strange thing is that it worked with firefly release, but it's kinda old
[22:14] <hemebond> Where can I find information on the radosgw/civetweb access log format?
[22:15] * osso (~osso@sgp01-1-78-233-150-179.fbx.proxad.net) has joined #ceph
[22:15] <proc_> TheSov3: if you had a guide for debian with hammer, please link it :)
[22:15] <proc_> alfredodeza: thanks anyway though!
[22:17] * proc__ (~proc@h-95-179.a463.priv.bahnhof.se) has joined #ceph
[22:18] * proc_ (~proc@h-95-179.a463.priv.bahnhof.se) Quit (Quit: leaving)
[22:18] * proc__ is now known as proc_
[22:25] * georgem (~Adium@206.108.127.16) Quit (Ping timeout: 480 seconds)
[22:25] * Maariu5_ (~AluAlu@4Z9AABWR5.tor-irc.dnsbl.oftc.net) Quit ()
[22:26] <vasu> alfredodeza: old verison of ceph-deploy but shouldn't matter (1.5.20)
[22:26] <vasu> in the above logs from proc_
[22:29] * mykola (~Mikolaj@91.225.201.130) Quit (Quit: away)
[22:35] * gregmark (~Adium@68.87.42.115) Quit (Quit: Leaving.)
[22:40] <SamYaple> had a power outage in a data center and my cluster went hard down. bringing it backup is proving troublesome. all of the osds are consuming a ton of ram, one of the osds was coumsing 65GB of ram, and i am running out of memory. I am looking for suggestion on how to proceed
[22:42] * linjan (~linjan@176.195.206.229) has joined #ceph
[22:42] * davidzlap (~Adium@2605:e000:1313:8003:7d53:ed75:bd58:a542) Quit (Read error: Connection reset by peer)
[22:42] * davidzlap (~Adium@2605:e000:1313:8003:7d53:ed75:bd58:a542) has joined #ceph
[22:43] * jayjay (~jjs@dsl-213-134-254-157.solcon.nl) has joined #ceph
[22:47] * baotiao (~baotiao@114.253.119.113) Quit (Quit: baotiao)
[22:51] * jayjay (~jjs@dsl-213-134-254-157.solcon.nl) Quit (Ping timeout: 480 seconds)
[22:52] * gregmark (~Adium@68.87.42.115) has joined #ceph
[22:55] * gregmark (~Adium@68.87.42.115) Quit ()
[22:57] * georgem (~Adium@184.151.190.225) has joined #ceph
[22:57] * georgem (~Adium@184.151.190.225) Quit ()
[22:58] * georgem (~Adium@206.108.127.16) has joined #ceph
[23:00] * tserong (~tserong@203-214-92-220.dyn.iinet.net.au) Quit (Ping timeout: 480 seconds)
[23:01] * haomaiwang (~haomaiwan@li715-221.members.linode.com) Quit (Remote host closed the connection)
[23:01] * haomaiwang (~haomaiwan@li715-221.members.linode.com) has joined #ceph
[23:09] * moore_ (~moore@71-211-73-118.phnx.qwest.net) has joined #ceph
[23:10] * TheSov2 (~TheSov@cip-248.trustwave.com) has joined #ceph
[23:10] <TheSov2> proc_, https://www.youtube.com/watch?v=po7SbQHdtV8
[23:10] <TheSov2> its step by step
[23:10] <TheSov2> you cannot go wrong with it
[23:11] <TheSov2> i wont even say good luck, you wont need it
[23:13] * TheSov3 (~TheSov@204.13.200.248) Quit (Ping timeout: 480 seconds)
[23:17] * moore (~moore@64.202.160.233) Quit (Ping timeout: 480 seconds)
[23:17] * brutuscat (~brutuscat@105.34.133.37.dynamic.jazztel.es) has joined #ceph
[23:23] * KaneK (~kane@12.206.204.58) has joined #ceph
[23:28] * SinZ|offline (~xENO_@93.174.93.101) has joined #ceph
[23:32] * linjan (~linjan@176.195.206.229) Quit (Ping timeout: 480 seconds)
[23:32] * georgem (~Adium@206.108.127.16) Quit (Quit: Leaving.)
[23:32] * vata1 (~vata@207.96.182.162) Quit (Quit: Leaving.)
[23:34] * KaneK (~kane@12.206.204.58) Quit (Quit: KaneK)
[23:34] * bene2 (~bene@nat-pool-bos-t.redhat.com) Quit (Quit: Konversation terminated!)
[23:38] * mhuang (~mhuang@119.254.120.72) has joined #ceph
[23:39] <TheSov2> is there any reason to allocate more than 5 gig to a osd journal
[23:42] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[23:42] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[23:43] * Exchizz (~kvirc@0x3ec6d532.inet.dsl.telianet.dk) has joined #ceph
[23:44] * jdillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) Quit (Quit: jdillaman)
[23:44] * daviddcc (~dcasier@80.12.59.12) Quit (Ping timeout: 480 seconds)
[23:45] * fedgoat (~theengine@45-31-177-36.lightspeed.austtx.sbcglobal.net) Quit (Quit: This computer has gone to sleep)
[23:45] * johnavp19891 (~jpetrini@8.39.115.8) Quit (Ping timeout: 480 seconds)
[23:48] <rkeene> TheSov2, It depends on how fast the backing device can consume data versus the journal
[23:51] <TheSov2> hey rkeene you were the one with the optimized system that was still going slow right
[23:51] * zaitcev (~zaitcev@c-50-130-189-82.hsd1.nm.comcast.net) has joined #ceph
[23:51] <TheSov2> i found out why my numbers looked better
[23:52] <TheSov2> u forgot to disable write caching when i rebuilt my cluster
[23:52] <TheSov2> err I*
[23:52] <TheSov2> then i got sad :(
[23:52] <TheSov2> anyway later
[23:52] * TheSov2 (~TheSov@cip-248.trustwave.com) Quit (Read error: Connection reset by peer)
[23:55] * mhuang (~mhuang@119.254.120.72) Quit (Quit: This computer has gone to sleep)
[23:55] * Ceph-Log-Bot (~logstash@185.66.248.215) has joined #ceph
[23:55] * Ceph-Log-Bot (~logstash@185.66.248.215) Quit (Read error: Connection reset by peer)
[23:56] * daviddcc (~dcasier@84.197.151.77.rev.sfr.net) has joined #ceph
[23:56] * garphy is now known as garphy`aw
[23:56] * redf_ (~red@chello080108089163.30.11.vie.surfer.at) has joined #ceph
[23:56] * mhuang (~mhuang@119.254.120.72) has joined #ceph
[23:58] * SinZ|offline (~xENO_@93.174.93.101) Quit ()

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.