#ceph IRC Log

Index

IRC Log for 2015-06-09

Timestamps are in GMT/BST.

[0:00] <doppelgrau> adrian15b: to baad idea in general, but I gues from the logs that you had soem PGs with only one copy (some operation that involes data migration + loss of one host ?) so IO got stuck since copies < min_size
[0:01] <doppelgrau> adrian15b: looks default, no fancy stuff there
[0:01] * blynch (~blynch@vm-nat.msi.umn.edu) has joined #ceph
[0:02] <doppelgrau> (not sure about the tunables, but the rule is default)
[0:02] <adrian15b> doppelgrau: What makes you say that I have some PGs with only one copy ? I mean I'm not questioning you. But how do you read that ? What string you are greping ?
[0:04] * linjan (~linjan@80.179.241.26) Quit (Ping timeout: 480 seconds)
[0:05] <doppelgrau> adrian15b: I was guessing, fits to the synthoms (slow request, undersized pgs) that resolves wehen you allowed the recovery
[0:06] * wushudoin_ (~wushudoin@38.140.108.2) Quit (Ping timeout: 480 seconds)
[0:06] <doppelgrau> adrian15b: if you had a pg dump while you had the problems, you could examine it in more detail
[0:07] * jbautista- (~wushudoin@transit-86-181-132-209.redhat.com) Quit (Ping timeout: 480 seconds)
[0:07] * cvstealth (~mcarey@pool-72-66-71-26.washdc.fios.verizon.net) Quit (Ping timeout: 480 seconds)
[0:08] * xarses (~andreww@12.164.168.117) Quit (Ping timeout: 480 seconds)
[0:14] * brad_mssw (~brad@66.129.88.50) Quit (Quit: Leaving)
[0:15] * QuantumBeep (~pakman__@9S0AAAU4U.tor-irc.dnsbl.oftc.net) Quit ()
[0:15] * csharp (~aldiyen@95.211.169.35) has joined #ceph
[0:16] * jbautista- (~wushudoin@38.140.108.2) has joined #ceph
[0:19] * jskinner (~jskinner@host-95-2-129.infobunker.com) Quit (Remote host closed the connection)
[0:19] <bjornar_> doppelgrau, its not about traffic
[0:20] <bjornar_> doppelgrau, 2 dc's are bad for quorum.. thats the problem
[0:20] <bjornar_> links are fine, dual 40G ~2km
[0:21] <doppelgrau> bjornar_: only one dc can win, else you would have a split brain situation
[0:21] <bjornar_> I know, thats why I want one passive
[0:21] * jashank42 (~jashan42@59.89.79.173) Quit (Ping timeout: 480 seconds)
[0:21] <bjornar_> so I end up with 2 active monitors in one dc and one active and one "passive" in the other...
[0:22] <bjornar_> I know this is probably not possible out of the box atm
[0:22] <doppelgrau> bjornar_: with 2 active and one passiv, the cluster would stop if one of the monitors fail
[0:23] <doppelgrau> bjornar_: with all three active the two in the firt dc still have the majority
[0:24] * jashank42 (~jashan42@59.89.79.173) has joined #ceph
[0:24] <doppelgrau> bjornar_: and if the first dc burn down, you might try the ???remove monitor from unhealty cluster??? so use the monitor (and the OSDs in the ???other??? dc) to get them up abgain: http://ceph.com/docs/master/rados/operations/add-or-rm-mons/
[0:25] * ircolle (~Adium@2601:1:a580:1735:805a:f446:f631:31c1) Quit (Quit: Leaving.)
[0:27] * diegows (~diegows@190.190.5.238) has joined #ceph
[0:29] * georgem (~Adium@fwnat.oicr.on.ca) Quit (Quit: Leaving.)
[0:31] * Nacer (~Nacer@2001:41d0:fe82:7200:25ea:e339:3796:2987) Quit (Ping timeout: 480 seconds)
[0:33] * Nacer (~Nacer@2001:41d0:fe82:7200:25ea:e339:3796:2987) has joined #ceph
[0:34] * mohan_ (~oftc-webi@103.27.8.44) has joined #ceph
[0:34] <mohan_> hi
[0:35] <mohan_> I was using qemu with ceph, on virsh start I got following error
[0:35] <mohan_> error: internal error: process exited while connecting to monitor: qemu-system-x86_64: -drive file=rbd:baadal_storage_pool/test_image:id=libvirt:key=AQAea3VV0JMSBxAA69dqyjn7zlfwlvybDgWbRA==:auth_supported=cephx\;none:mon_host=10.237.23.132\:6789,if=none,id=drive-ide0-0-1: error reading header from test_image
[0:38] * marrusl (~mark@cpe-24-90-46-248.nyc.res.rr.com) Quit (Quit: bye!)
[0:42] * wschulze1 (~wschulze@cpe-69-206-240-164.nyc.res.rr.com) has joined #ceph
[0:44] * wschulze (~wschulze@cpe-69-206-240-164.nyc.res.rr.com) Quit (Read error: Connection reset by peer)
[0:44] * wschulze (~wschulze@cpe-69-206-240-164.nyc.res.rr.com) has joined #ceph
[0:45] * csharp (~aldiyen@9S0AAAU58.tor-irc.dnsbl.oftc.net) Quit ()
[0:46] * delcake (~spate@nx-01.tor-exit.network) has joined #ceph
[0:50] <Tene> Hey, just checking to see whether I'm misinterpreting some release notes properly or not, is Hammer actually released and production ready, or still release candidate? The v0.94 release notes say "This major release is expected to form the basis of the next long-term stable series.", but I can't quite tell if that's saying that it *is* the next long-term release, or that it *will* be later.
[0:51] * wschulze1 (~wschulze@cpe-69-206-240-164.nyc.res.rr.com) Quit (Ping timeout: 480 seconds)
[0:52] <dmick> it's the LTS release
[0:52] <dmick> note there is a v0.94.1
[0:53] * sjm (~sjm@pool-173-70-76-86.nwrknj.fios.verizon.net) has left #ceph
[0:53] <Tene> dmick: Thanks.
[0:53] * dneary (~dneary@nat-pool-bos-u.redhat.com) Quit (Ping timeout: 480 seconds)
[0:55] * alram (~alram@192.41.52.12) Quit (Ping timeout: 480 seconds)
[0:55] * Concubidated1 (~Adium@192.41.52.12) Quit (Ping timeout: 480 seconds)
[0:59] * stiopa (~stiopa@cpc73828-dals21-2-0-cust630.20-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[1:11] * blynch (~blynch@vm-nat.msi.umn.edu) Quit (Remote host closed the connection)
[1:12] * Concubidated (~Adium@66.87.126.101) has joined #ceph
[1:13] * BManojlovic (~steki@cable-89-216-223-146.dynamic.sbb.rs) Quit (Quit: Ja odoh a vi sta 'ocete...)
[1:15] * alram (~alram@64.134.221.151) has joined #ceph
[1:15] * delcake (~spate@5NZAADG6J.tor-irc.dnsbl.oftc.net) Quit ()
[1:15] * LRWerewolf (~Azerothia@9S0AAAU8G.tor-irc.dnsbl.oftc.net) has joined #ceph
[1:31] * janos_ (~messy@static-71-176-211-4.rcmdva.fios.verizon.net) has joined #ceph
[1:32] * moore_ (~moore@64.202.160.88) Quit (Remote host closed the connection)
[1:33] * fam_away is now known as fam
[1:35] * lcurtis (~lcurtis@47.19.105.250) Quit (Quit: Ex-Chat)
[1:45] * jdillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) Quit (Quit: jdillaman)
[1:45] * LRWerewolf (~Azerothia@9S0AAAU8G.tor-irc.dnsbl.oftc.net) Quit ()
[1:45] * Jaska (~pico@exit1.ipredator.se) has joined #ceph
[1:56] * bjornar_ (~bjornar@93.187.84.175) Quit (Ping timeout: 480 seconds)
[2:04] * oms101 (~oms101@p20030057EA0B3000C6D987FFFE4339A1.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[2:08] * snerd_ is now known as snerd
[2:11] * cholcombe (~chris@c-73-180-29-35.hsd1.or.comcast.net) Quit (Ping timeout: 480 seconds)
[2:12] * oms101 (~oms101@p20030057EA0B2000C6D987FFFE4339A1.dip0.t-ipconnect.de) has joined #ceph
[2:15] <jidar> hey guys, I'm trying to deploy calamari/inktank on RHEL 7 but I can't find the ICE-1.2.2 rhel7 tar.gz
[2:15] * Jaska (~pico@2FBAAB9EG.tor-irc.dnsbl.oftc.net) Quit ()
[2:15] * delcake (~Heliwr@chomsky.torservers.net) has joined #ceph
[2:15] <jidar> that this doc references, any ideas?
[2:17] * mohan_ (~oftc-webi@103.27.8.44) Quit (Quit: Page closed)
[2:18] * owasserm (~owasserm@206.169.83.146) Quit (Ping timeout: 480 seconds)
[2:19] <adrian15b> How do you inject an empy string in "ceph tell osd.* injectargs" ? Thank you.
[2:19] * joao (~joao@249.38.136.95.rev.vodafone.pt) Quit (Quit: Leaving)
[2:24] <adrian15b> I'm having problems connecting to OSDs or to monitors, not sure. http://paste.debian.net/213809/ . How do I solve them so that I can injectargs finally ? Thank you.
[2:25] <adrian15b> According to several status both monitors and OSD are up and running but maybe there's something other.
[2:26] * jwilkins (~jwilkins@2601:9:4580:f4c:ea2a:eaff:fe08:3f1d) Quit (Quit: Leaving)
[2:26] <adrian15b> No ceph log moving. I guess something went wrong when injectargs with "". Trying to restart monitors.
[2:32] <adrian15b> This is not working at all :(.
[2:33] * RomeroJnr (~h0m3r@hosd.leaseweb.net) Quit ()
[2:33] <adrian15b> Why ceph is so sensible ?
[2:35] <adrian15b> Else is there any place where I can un-inject args manually ? By editing some xml or something ? Thank you.??
[2:37] * arbrandes (~arbrandes@177.138.114.132) Quit (Quit: Leaving)
[2:38] * vpol (~vpol@000131a0.user.oftc.net) has joined #ceph
[2:38] * elder (~elder@c-24-245-18-91.hsd1.mn.comcast.net) Quit (Quit: Leaving)
[2:40] * elder (~elder@c-24-245-18-91.hsd1.mn.comcast.net) has joined #ceph
[2:40] * ChanServ sets mode +o elder
[2:41] * Concubidated (~Adium@66.87.126.101) Quit (Read error: Connection reset by peer)
[2:41] <dmick> jidar: ICE/Red Hat Ceph Storage is a licensed product
[2:42] <jidar> yes, and I have the licenses for this
[2:42] <dmick> where are you trying to find the release?
[2:43] <jidar> yum search
[2:43] <jidar> after I enable all of the calamari/ceph repos
[2:43] <dmick> you're searching for a tar.gz with yum?
[2:43] <jidar> I'm searching for ice, and all I have is ice_setup
[2:44] <jidar> From repo : rhel-7-server-rhceph-1.2-installer-rpms
[2:44] <jidar> Summary : Red Hat Ceph Storage setup tool
[2:44] <dmick> that repo would not contain a tar.gz; you would have gotten that from a direct download. That repo is for 1.2.3 and later, which are licensed separately through Red Hat
[2:44] <jidar> but this doesn't include how to setup the repos, nor what packages you need to drop into the repo directory, nor how one might go about that (I assume this is part of the ICE tar.gz)
[2:45] * delcake (~Heliwr@5NZAADHAJ.tor-irc.dnsbl.oftc.net) Quit ()
[2:45] * Swompie` (~bildramer@195.169.125.226) has joined #ceph
[2:46] <dmick> you would never, as a user/installer, "drop" packages "into" a repo dir
[2:46] <jidar> the ice_setup process would have you believe that
[2:46] <dmick> 1.2.2 was distributed as a tarball to registered customers directly from a secure website
[2:46] <adrian15b> Seems that rebooting the ceph nodes one per one solves my problem.
[2:46] <jidar> I found that website but it requires a username/password
[2:46] <dmick> 1.2.3 was distributed as an ISO to registered customers
[2:47] <jidar> dmick: do you happen to know what licensing option I need from redhat to accomplish this?
[2:47] <dmick> actually 1.2.3 is not yet released.
[2:47] <jidar> then what license option I require for 1.2.2
[2:47] <dmick> jidar: I do not, specifically, but your account rep should be able to help
[2:48] <jidar> yea, they are slow though :(
[2:48] <dmick> yeah. sorry, that's the other side of the house from me
[2:48] <jidar> Its taken me weeks to get all of this situated into my NFR copies in the first place
[2:48] * dyasny (~dyasny@198.251.58.23) Quit (Ping timeout: 480 seconds)
[2:48] <jidar> plus my customer is waiting and already bough what they thought they needed to buy
[2:48] <dmick> NFR?
[2:48] <jidar> I'm a consultant
[2:49] <jidar> we're partnered with redhat
[2:49] <jidar> NFR stands for Not for Resell
[2:49] <dmick> ok
[2:50] <jidar> thank you for the heads up though, now at least I don't have to open a support ticket :)
[2:50] <dmick> https://access.redhat.com/products/red-hat-ceph-storage
[2:51] <jidar> ahh yea, it tells me I need an active subscription etc
[2:52] <dmick> yeah, you need to talk to the sales/partner management folks
[2:52] <jidar> emailing now :)
[2:54] * kefu (~kefu@119.254.120.68) has joined #ceph
[2:54] * kefu (~kefu@119.254.120.68) Quit ()
[3:02] <jidar> https://access.redhat.com/beta/documentation/en/red-hat-ceph-storage-123-installation-guide-for-rhel-x86-64#setting-up-your-administration-server
[3:02] <TheSov> oh god just playing with ceph is so awesome!
[3:02] * yguang11 (~yguang11@2001:4998:effd:600:901e:438f:212f:6224) Quit (Remote host closed the connection)
[3:08] * yanzheng (~zhyan@171.216.94.129) has joined #ceph
[3:11] * Kupo1 (~tyler.wil@23.111.254.159) has joined #ceph
[3:11] * Kupo1 (~tyler.wil@23.111.254.159) has left #ceph
[3:12] * diegows (~diegows@190.190.5.238) Quit (Ping timeout: 480 seconds)
[3:15] * Swompie` (~bildramer@9S0AAAVB1.tor-irc.dnsbl.oftc.net) Quit ()
[3:15] * Jaska (~Shadow386@exit1.ipredator.se) has joined #ceph
[3:20] * bandrus (~brian@198.8.80.168) Quit (Quit: Leaving.)
[3:21] <jidar> dmick: ok, I had access but for whatever reason you really had to download that ISO
[3:21] <jidar> goofy!
[3:22] <dmick> I guess it's released then. humpf.
[3:22] <dmick> ok
[3:22] <dmick> (the intent is to support totally-standalone installations, and the repo setup is ... nonstandard as a result)
[3:22] <jidar> yea, I gathered that
[3:22] <dmick> it's evolving
[3:23] <jidar> also, the docs I linked to a minute ago are soooooo much better than what's in this reference arch
[3:26] <dmick> taking your word for it.
[3:26] <dmick> don't even know what a 'reference arch' might be
[3:26] <jidar> so redhat came out with a reference arch about two or three months ago for building a HA Openstack cluster plus HA ceph, etc
[3:27] <jidar> it's got 3 controllers and 3 storage nodes, plus any number of compute hosts
[3:27] <jidar> some parts of the doc are sparse,
[3:28] <dmick> well that also includes openstack
[3:29] <dmick> which is somewhat less than trivial
[3:29] <dmick> (more than trivial?)
[3:31] <jidar> I think it's a bit more than trivial :)
[3:34] * KevinPerks (~Adium@cpe-75-177-32-14.triad.res.rr.com) Quit (Quit: Leaving.)
[3:34] * ira (~ira@c-71-233-225-22.hsd1.ma.comcast.net) Quit (Ping timeout: 480 seconds)
[3:35] * jclm (~jclm@122.181.21.134) Quit (Quit: Leaving.)
[3:37] * ira (~ira@c-71-233-225-22.hsd1.ma.comcast.net) has joined #ceph
[3:40] * ganesh (~oftc-webi@103.27.8.44) has joined #ceph
[3:40] <ganesh> hi
[3:41] * kefu (~kefu@119.254.120.68) has joined #ceph
[3:41] <ganesh> is layering enabled in rbd image created by qemu-img
[3:42] * KevinPerks (~Adium@cpe-75-177-32-14.triad.res.rr.com) has joined #ceph
[3:43] * ira (~ira@c-71-233-225-22.hsd1.ma.comcast.net) Quit (Quit: Leaving)
[3:45] * Jaska (~Shadow386@9S0AAAVDE.tor-irc.dnsbl.oftc.net) Quit ()
[4:01] * bitserker (~toni@188.87.126.203) Quit (Quit: Leaving.)
[4:12] * shang (~ShangWu@175.41.48.77) has joined #ceph
[4:15] * BlS (~Dragonsha@89.105.194.73) has joined #ceph
[4:18] * flisky (~Thunderbi@106.39.60.34) has joined #ceph
[4:19] * georgem (~Adium@206-248-174-223.dsl.teksavvy.com) has joined #ceph
[4:20] * nsoffer (~nsoffer@bzq-79-180-80-86.red.bezeqint.net) Quit (Ping timeout: 480 seconds)
[4:21] * DV (~veillard@2001:41d0:a:f29f::1) Quit (Ping timeout: 480 seconds)
[4:22] * zhaochao (~zhaochao@125.39.8.226) has joined #ceph
[4:26] * shang (~ShangWu@175.41.48.77) Quit (Quit: Ex-Chat)
[4:26] * shang (~ShangWu@175.41.48.77) has joined #ceph
[4:33] * DV (~veillard@2001:41d0:1:d478::1) has joined #ceph
[4:34] * owasserm (~owasserm@216.1.187.164) has joined #ceph
[4:34] * Concubidated (~Adium@66-87-127-8.pools.spcsdns.net) has joined #ceph
[4:35] * tmh_ (~e@loophole.cc) Quit (Read error: Connection reset by peer)
[4:35] * yguang11 (~yguang11@12.31.82.125) has joined #ceph
[4:35] * tmh_ (~e@loophole.cc) has joined #ceph
[4:35] * Concubidated1 (~Adium@66-87-127-8.pools.spcsdns.net) has joined #ceph
[4:35] * Concubidated (~Adium@66-87-127-8.pools.spcsdns.net) Quit (Read error: Connection reset by peer)
[4:36] * yguang11_ (~yguang11@2001:4998:effd:7801::10be) has joined #ceph
[4:43] * yguang11 (~yguang11@12.31.82.125) Quit (Ping timeout: 480 seconds)
[4:43] * vpol (~vpol@000131a0.user.oftc.net) Quit (Quit: vpol)
[4:43] * bitserker (~toni@213.229.187.102) has joined #ceph
[4:45] * georgem (~Adium@206-248-174-223.dsl.teksavvy.com) Quit (Quit: Leaving.)
[4:45] * BlS (~Dragonsha@9S0AAAVFY.tor-irc.dnsbl.oftc.net) Quit ()
[4:46] * thundercloud (~bildramer@exit1.telostor.ca) has joined #ceph
[4:50] * vpol (~vpol@000131a0.user.oftc.net) has joined #ceph
[4:53] * Nacer_ (~Nacer@203-206-190-109.dsl.ovh.fr) has joined #ceph
[4:53] * Nacer (~Nacer@2001:41d0:fe82:7200:25ea:e339:3796:2987) Quit (Read error: Connection reset by peer)
[4:53] * vbellur (~vijay@122.172.46.139) has joined #ceph
[5:00] * fam is now known as fam_away
[5:01] * CAPSLOCK2000 (~oftc@2001:984:3be3:1::8) Quit (Ping timeout: 480 seconds)
[5:02] * overclk (~overclk@121.244.87.117) has joined #ceph
[5:05] * peppe1 (~peppe1@198.24.6.220) has joined #ceph
[5:05] * wschulze (~wschulze@cpe-69-206-240-164.nyc.res.rr.com) Quit (Read error: Connection reset by peer)
[5:05] * wschulze (~wschulze@cpe-69-206-240-164.nyc.res.rr.com) has joined #ceph
[5:05] * vpol (~vpol@000131a0.user.oftc.net) Quit (Quit: vpol)
[5:06] * kklimonda_ (sid72883@id-72883.highgate.irccloud.com) Quit (Remote host closed the connection)
[5:06] * supay (sid47179@id-47179.highgate.irccloud.com) Quit (Remote host closed the connection)
[5:06] * carmstrong (sid22558@id-22558.highgate.irccloud.com) Quit (Remote host closed the connection)
[5:06] * kamalmarhubi (sid26581@id-26581.highgate.irccloud.com) Quit (Remote host closed the connection)
[5:07] * gabrtv (sid36209@id-36209.brockwell.irccloud.com) Quit (Remote host closed the connection)
[5:08] <peppe1> hi folks, i have a wrking ceph cluster running on centos7 though could not install calamari (more specifically diamond as requires python 2.6 and my cluster is on 2.7) as the vagrant box @ github is based on centos64. I was wondering if anyone can help?
[5:13] * midnightrunner (~midnightr@216.113.160.71) Quit (Remote host closed the connection)
[5:15] * xinxinsh (~xinxinsh@shzdmzpr02-ext.sh.intel.com) has joined #ceph
[5:15] * thundercloud (~bildramer@0SGAAA99F.tor-irc.dnsbl.oftc.net) Quit ()
[5:18] * kefu (~kefu@119.254.120.68) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[5:20] * jashank42 (~jashan42@59.89.79.173) Quit (Ping timeout: 480 seconds)
[5:21] * Vacuum_ (~Vacuum@i59F790E8.versanet.de) has joined #ceph
[5:22] * jclm (~jclm@121.244.87.117) has joined #ceph
[5:27] * squ (~Thunderbi@46.109.36.167) has joined #ceph
[5:28] * Vacuum (~Vacuum@i59F79F2F.versanet.de) Quit (Ping timeout: 480 seconds)
[5:29] * xarses (~andreww@50.141.6.2) has joined #ceph
[5:30] * jashank42 (~jashan42@117.214.199.167) has joined #ceph
[5:31] * yguang11_ (~yguang11@2001:4998:effd:7801::10be) Quit (Ping timeout: 480 seconds)
[5:38] * jclm1 (~jclm@121.244.87.117) has joined #ceph
[5:42] * yguang11 (~yguang11@2001:4998:effd:7801::10be) has joined #ceph
[5:42] * bitserker (~toni@213.229.187.102) Quit (Ping timeout: 480 seconds)
[5:42] * jclm (~jclm@121.244.87.117) Quit (Ping timeout: 480 seconds)
[5:45] * kiasyn (~Da_Pineap@vmd7136.contabo.host) has joined #ceph
[5:55] * xarses (~andreww@50.141.6.2) Quit (Remote host closed the connection)
[5:56] * alram (~alram@64.134.221.151) Quit (Ping timeout: 480 seconds)
[5:59] <TheSov> peppe1, are u installing calamari for a reason? the latest version is hammer
[5:59] <peppe1> just exploring :) I already have a working ceph cluster based on centos7 (python 2.7)
[5:59] <peppe1> https://ceph.com/category/calamari/
[6:00] <peppe1> followed the above to build calamari-server and client and most worked ...server is up though diamon rpm could not be installed on centos7 due to python 2.6 dependencies
[6:00] <peppe1> @github the vagrant box is based on centos64
[6:00] <cephalobot`> peppe1: Error: "github" is not a valid command.
[6:01] <peppe1> so, need some help to get diamond rpm built that can be installed on centos7/python2.7
[6:03] * wkennington (~william@76.77.181.17) Quit (Remote host closed the connection)
[6:04] * OutOfNoWhere (~rpb@199.68.195.102) Quit (Ping timeout: 480 seconds)
[6:08] * wkennington (~william@76.77.181.17) has joined #ceph
[6:10] * yguang11 (~yguang11@2001:4998:effd:7801::10be) Quit (Ping timeout: 480 seconds)
[6:10] * bitserker (~toni@188.87.126.203) has joined #ceph
[6:10] * yguang11 (~yguang11@2001:4998:effd:7801::10be) has joined #ceph
[6:13] <adrian15b> These commands have solved my virtual machines I/O problems (and using a journal disk which I wasn't used before). http://paste.debian.net/214463/ Is there anything more that I could tweak to give more I/O to normal ceph usage? Thank you.
[6:15] * kiasyn (~Da_Pineap@9S0AAAVJW.tor-irc.dnsbl.oftc.net) Quit ()
[6:15] * WedTM (~Nephyrin@91.219.236.218) has joined #ceph
[6:17] * bitserker (~toni@188.87.126.203) Quit (Quit: Leaving.)
[6:17] * bitserker (~toni@188.87.126.203) has joined #ceph
[6:24] * derjohn_mob (~aj@p578b6aa1.dip0.t-ipconnect.de) has joined #ceph
[6:27] <darkfader> adrian15b: fyi the 'idle' io scheduler class has no prios
[6:27] <darkfader> iirc
[6:32] * kanagaraj (~kanagaraj@121.244.87.117) has joined #ceph
[6:32] * jashank42 (~jashan42@117.214.199.167) Quit (Ping timeout: 480 seconds)
[6:34] * janos_ (~messy@static-71-176-211-4.rcmdva.fios.verizon.net) Quit (Read error: Connection reset by peer)
[6:34] * janos_ (~messy@static-71-176-211-4.rcmdva.fios.verizon.net) has joined #ceph
[6:39] * amote (~amote@121.244.87.116) has joined #ceph
[6:41] * xinxinsh (~xinxinsh@shzdmzpr02-ext.sh.intel.com) Quit (Ping timeout: 480 seconds)
[6:42] * jashank42 (~jashan42@117.214.255.78) has joined #ceph
[6:45] * WedTM (~Nephyrin@9S0AAAVLL.tor-irc.dnsbl.oftc.net) Quit ()
[6:45] * Pirate (~Arcturus@195.169.125.226) has joined #ceph
[6:46] <Tene> I'm trying to create an erasure coded pool. I've got 8 hosts with 3 OSDs each, and I created a pool with k=8 m=3. All my pgs are stuck inactive and unclean. ceph health detail shows the following for every pg in the pool:
[6:46] <Tene> pg 16.188 is stuck unclean since forever, current state creating, last acting []
[6:46] <Tene> But, they're not actually getting created.
[6:47] <Tene> Any ideas?
[6:48] <Tene> Oh, hm, ruleset-failure-domain=host
[6:49] * wschulze (~wschulze@cpe-69-206-240-164.nyc.res.rr.com) Quit (Quit: Leaving.)
[6:50] <Tene> I get the same problem if I set ruleset-failure-domain=osd
[6:55] <Tene> Ahh, nevermind, I think I found the problem.
[6:57] * vikhyat (~vumrao@121.244.87.116) has joined #ceph
[7:01] * MACscr (~Adium@2601:d:c800:de3:439:e533:783b:f33e) Quit (Remote host closed the connection)
[7:01] * bitserker (~toni@188.87.126.203) Quit (Read error: No route to host)
[7:03] * gaveen (~gaveen@175.157.42.190) has joined #ceph
[7:04] * adrian15b (~kvirc@31.Red-79-159-37.dynamicIP.rima-tde.net) Quit (Ping timeout: 480 seconds)
[7:05] <Tene> I had assigned it a ruleset with type 'replicated'
[7:05] * branto (~branto@ip-213-220-214-203.net.upcbroadband.cz) has joined #ceph
[7:09] * peppe1 (~peppe1@198.24.6.220) Quit (Quit: Leaving...)
[7:11] * debian112 (~bcolbert@24.126.201.64) Quit (Quit: Leaving.)
[7:12] * KevinPerks (~Adium@cpe-75-177-32-14.triad.res.rr.com) Quit (Quit: Leaving.)
[7:13] * treenerd (~treenerd@178.115.132.254.wireless.dyn.drei.com) has joined #ceph
[7:15] * xinxinsh (~xinxinsh@shzdmzpr02-ext.sh.intel.com) has joined #ceph
[7:15] * Pirate (~Arcturus@9S0AAAVM6.tor-irc.dnsbl.oftc.net) Quit ()
[7:16] * Quackie (~spate@sh3llbox.hu) has joined #ceph
[7:21] * vbellur (~vijay@122.172.46.139) Quit (Ping timeout: 480 seconds)
[7:26] * yguang11 (~yguang11@2001:4998:effd:7801::10be) Quit (Ping timeout: 480 seconds)
[7:31] * yguang11 (~yguang11@2001:4998:effd:7801::10be) has joined #ceph
[7:35] * xarses (~andreww@50.141.6.2) has joined #ceph
[7:35] * treenerd (~treenerd@178.115.132.254.wireless.dyn.drei.com) Quit (Ping timeout: 480 seconds)
[7:37] * rdas (~rdas@121.244.87.116) has joined #ceph
[7:38] * treenerd (~treenerd@178.115.132.254.wireless.dyn.drei.com) has joined #ceph
[7:45] * MACscr (~Adium@2601:d:c800:de3:c00c:d829:59fb:6e0d) has joined #ceph
[7:45] * Quackie (~spate@9S0AAAVOY.tor-irc.dnsbl.oftc.net) Quit ()
[7:45] * raindog (~N3X15@91.219.236.218) has joined #ceph
[7:46] * treenerd (~treenerd@178.115.132.254.wireless.dyn.drei.com) Quit (Ping timeout: 480 seconds)
[7:46] * shylesh (~shylesh@121.244.87.124) has joined #ceph
[7:49] * xarses (~andreww@50.141.6.2) Quit (Remote host closed the connection)
[7:52] * jashank42 (~jashan42@117.214.255.78) Quit (Ping timeout: 480 seconds)
[7:53] * xarses (~andreww@50.141.6.2) has joined #ceph
[8:00] * karnan (~karnan@121.244.87.117) has joined #ceph
[8:01] * jashank42 (~jashan42@59.89.62.34) has joined #ceph
[8:02] * kefu (~kefu@119.254.120.68) has joined #ceph
[8:03] * kefu (~kefu@119.254.120.68) Quit ()
[8:03] * vbellur (~vijay@121.244.87.117) has joined #ceph
[8:04] * kefu (~kefu@119.254.120.68) has joined #ceph
[8:06] * kefu (~kefu@119.254.120.68) Quit ()
[8:07] <TheSov> help i go throuh the process of ceph-deploy and everything runs good until i do the gatherkeys, then it craps out
[8:07] <TheSov> [ceph_deploy.gatherkeys][WARNIN] Unable to find /etc/ceph/ceph.client.admin.keyring on ceph-mon1
[8:07] * Concubidated1 is now known as Concubidated
[8:08] * haomaiwang (~haomaiwan@114.111.166.250) Quit (Remote host closed the connection)
[8:10] * gaveen (~gaveen@175.157.42.190) Quit (Ping timeout: 480 seconds)
[8:11] * vata (~vata@cable-21.246.173-197.electronicbox.net) Quit (Quit: Leaving.)
[8:13] * TMM (~hp@178-84-46-106.dynamic.upc.nl) Quit (Quit: Ex-Chat)
[8:15] * derjohn_mob (~aj@p578b6aa1.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[8:15] * raindog (~N3X15@8Q4AABDXA.tor-irc.dnsbl.oftc.net) Quit ()
[8:15] * redbeast12 (~Phase@exit1.ipredator.se) has joined #ceph
[8:17] * sleinen (~Adium@macsl.switch.ch) has joined #ceph
[8:17] * Sysadmin88 (~IceChat77@054527d3.skybroadband.com) Quit (Quit: If at first you don't succeed, skydiving is not for you)
[8:22] * cooldharma06 (~chatzilla@14.139.180.40) has joined #ceph
[8:24] * stiopa (~stiopa@cpc73828-dals21-2-0-cust630.20-2.cable.virginm.net) has joined #ceph
[8:26] * bitserker (~toni@188.87.126.203) has joined #ceph
[8:29] * carmstrong (sid22558@id-22558.highgate.irccloud.com) has joined #ceph
[8:33] * vpol (~vpol@000131a0.user.oftc.net) has joined #ceph
[8:35] * kamalmarhubi (sid26581@id-26581.highgate.irccloud.com) has joined #ceph
[8:35] * haomaiwang (~haomaiwan@182.48.117.114) has joined #ceph
[8:37] * cok (~chk@2a02:2350:18:1010:e826:21c4:abff:168) has joined #ceph
[8:39] * gaveen (~gaveen@175.157.42.190) has joined #ceph
[8:39] * Nacer_ (~Nacer@203-206-190-109.dsl.ovh.fr) Quit (Remote host closed the connection)
[8:39] * yguang11 (~yguang11@2001:4998:effd:7801::10be) Quit (Ping timeout: 480 seconds)
[8:39] * CAPSLOCK2000 (~oftc@2001:984:3be3:1::8) has joined #ceph
[8:45] * redbeast12 (~Phase@9S0AAAVS5.tor-irc.dnsbl.oftc.net) Quit ()
[8:45] * visored (~LRWerewol@85.143.95.50) has joined #ceph
[8:48] * gabrtv (sid36209@id-36209.brockwell.irccloud.com) has joined #ceph
[8:50] * jashank42 (~jashan42@59.89.62.34) Quit (Ping timeout: 480 seconds)
[8:50] * ade (~abradshaw@tmo-100-250.customers.d1-online.com) has joined #ceph
[8:52] * TMM (~hp@sams-office-nat.tomtomgroup.com) has joined #ceph
[8:53] * Hemanth (~Hemanth@121.244.87.117) has joined #ceph
[8:53] * devicenull (sid4013@id-4013.charlton.irccloud.com) Quit (Remote host closed the connection)
[8:53] * scalability-junk (sid6422@id-6422.charlton.irccloud.com) Quit (Remote host closed the connection)
[8:53] * getzburg (sid24913@id-24913.charlton.irccloud.com) Quit (Remote host closed the connection)
[8:53] * Pintomatic (sid25118@id-25118.charlton.irccloud.com) Quit (Remote host closed the connection)
[8:53] * immesys (sid44615@id-44615.charlton.irccloud.com) Quit (Remote host closed the connection)
[8:53] * JohnPreston78 (sid31393@id-31393.charlton.irccloud.com) Quit (Remote host closed the connection)
[8:53] * treenerd (~treenerd@85.193.140.98) has joined #ceph
[8:54] * b0e (~aledermue@213.95.25.82) has joined #ceph
[8:54] * treenerd (~treenerd@85.193.140.98) Quit ()
[8:57] * adrian15b (~kvirc@btactic.ddns.jazztel.es) has joined #ceph
[8:59] * jashank42 (~jashan42@59.89.60.43) has joined #ceph
[8:59] * Be-El (~quassel@fb08-bcf-pc01.computational.bio.uni-giessen.de) has joined #ceph
[8:59] <Be-El> hi
[8:59] * dugravot6 (~dugravot6@dn-infra-04.lionnois.univ-lorraine.fr) has joined #ceph
[9:00] * supay (sid47179@id-47179.highgate.irccloud.com) has joined #ceph
[9:02] * visored (~LRWerewol@8Q4AABDZA.tor-irc.dnsbl.oftc.net) Quit (Remote host closed the connection)
[9:02] * Chrissi_ (~Kottizen@edwardsnowden2.torservers.net) has joined #ceph
[9:03] * derjohn_mob (~aj@fw.gkh-setu.de) has joined #ceph
[9:05] * kefu (~kefu@119.254.120.68) has joined #ceph
[9:05] * devicenull (sid4013@id-4013.charlton.irccloud.com) has joined #ceph
[9:07] * T1w (~jens@node3.survey-it.dk) has joined #ceph
[9:08] * scalability-junk (sid6422@id-6422.charlton.irccloud.com) has joined #ceph
[9:10] * badone (~brad@CPE-121-215-241-179.static.qld.bigpond.net.au) Quit (Ping timeout: 480 seconds)
[9:12] * dugravot61 (~dugravot6@nat-persul-plg.wifi.univ-lorraine.fr) has joined #ceph
[9:14] * vikhyat (~vumrao@121.244.87.116) Quit (Remote host closed the connection)
[9:14] * dugravot61 (~dugravot6@nat-persul-plg.wifi.univ-lorraine.fr) Quit ()
[9:16] * dugravot61 (~dugravot6@nat-persul-plg.wifi.univ-lorraine.fr) has joined #ceph
[9:16] * bitserker (~toni@188.87.126.203) Quit (Quit: Leaving.)
[9:17] * stiopa (~stiopa@cpc73828-dals21-2-0-cust630.20-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[9:18] * rotbeard (~redbeard@2a02:908:df10:d300:76f0:6dff:fe3b:994d) Quit (Quit: Verlassend)
[9:18] * SkyEye (~gaveen@175.157.54.12) has joined #ceph
[9:19] * dugravot6 (~dugravot6@dn-infra-04.lionnois.univ-lorraine.fr) Quit (Ping timeout: 480 seconds)
[9:19] * jashank42 (~jashan42@59.89.60.43) Quit (Ping timeout: 480 seconds)
[9:21] * gaveen (~gaveen@175.157.42.190) Quit (Ping timeout: 480 seconds)
[9:21] * vikhyat (~vumrao@121.244.87.116) has joined #ceph
[9:21] * shnarch (~shnarch@bzq-109-67-128-59.red.bezeqint.net) Quit (Ping timeout: 480 seconds)
[9:23] * thomnico (~thomnico@2a01:e35:8b41:120:a996:3c09:b684:4b7a) has joined #ceph
[9:23] * kklimonda_ (sid72883@id-72883.highgate.irccloud.com) has joined #ceph
[9:26] * getzburg (sid24913@id-24913.charlton.irccloud.com) has joined #ceph
[9:26] * Pintomatic (sid25118@id-25118.charlton.irccloud.com) has joined #ceph
[9:27] * wicope (~wicope@0001fd8a.user.oftc.net) has joined #ceph
[9:28] * schmee (~quassel@phobos.isoho.st) Quit (Remote host closed the connection)
[9:28] * espeer (~quassel@phobos.isoho.st) Quit (Remote host closed the connection)
[9:29] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[9:29] * analbeard (~shw@support.memset.com) has joined #ceph
[9:29] * analbeard (~shw@support.memset.com) has left #ceph
[9:29] * linjan (~linjan@95.35.28.241) has joined #ceph
[9:29] * jashank42 (~jashan42@117.214.250.47) has joined #ceph
[9:30] * espeer (~quassel@phobos.isoho.st) has joined #ceph
[9:30] * schmee (~quassel@41.78.129.253) has joined #ceph
[9:30] * linjan (~linjan@95.35.28.241) Quit (Read error: No route to host)
[9:31] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[9:31] * JohnPreston78 (sid31393@id-31393.charlton.irccloud.com) has joined #ceph
[9:31] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[9:31] * jashank42 (~jashan42@117.214.250.47) Quit ()
[9:32] * thomnico (~thomnico@2a01:e35:8b41:120:a996:3c09:b684:4b7a) Quit (Ping timeout: 480 seconds)
[9:32] * Chrissi_ (~Kottizen@8Q4AABDZX.tor-irc.dnsbl.oftc.net) Quit ()
[9:37] * topro (~prousa@host-62-245-142-50.customer.m-online.net) Quit (Quit: Konversation terminated!)
[9:39] * kefu (~kefu@119.254.120.68) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[9:40] * dgurtner (~dgurtner@178.197.231.155) has joined #ceph
[9:40] * topro (~prousa@host-62-245-142-50.customer.m-online.net) has joined #ceph
[9:40] * immesys (sid44615@id-44615.charlton.irccloud.com) has joined #ceph
[9:47] * Nacer (~Nacer@252-87-190-213.intermediasud.com) has joined #ceph
[9:51] * vpol (~vpol@000131a0.user.oftc.net) Quit (Quit: vpol)
[9:55] * haomaiwa_ (~haomaiwan@182.48.117.114) has joined #ceph
[9:55] * haomaiwang (~haomaiwan@182.48.117.114) Quit (Read error: Connection reset by peer)
[9:57] * kawa2014 (~kawa@89.184.114.246) has joined #ceph
[10:03] * thomnico (~thomnico@2a01:e35:8b41:120:a996:3c09:b684:4b7a) has joined #ceph
[10:05] * haomaiwa_ (~haomaiwan@182.48.117.114) Quit (Remote host closed the connection)
[10:12] * circ-user-ClZ7Y (~circuser-@91.241.13.28) has joined #ceph
[10:16] * shnarch (~shnarch@2.52.61.121) has joined #ceph
[10:20] * antoine (~bourgault@192.93.37.4) has joined #ceph
[10:20] * jordanP (~jordan@scality-jouf-2-194.fib.nerim.net) has joined #ceph
[10:20] * haomaiwang (~haomaiwan@114.111.166.250) has joined #ceph
[10:20] * ledgr (~qstion@37.157.144.44) has joined #ceph
[10:20] * jashank42 (~jashan42@117.214.250.47) has joined #ceph
[10:24] * lucas1 (~Thunderbi@218.76.52.64) Quit (Remote host closed the connection)
[10:24] * jklare (~jklare@185.27.181.36) has joined #ceph
[10:26] * linjan (~linjan@195.110.41.9) has joined #ceph
[10:26] * linjan (~linjan@195.110.41.9) Quit (Read error: Connection reset by peer)
[10:33] * Grimmer (~Chrissi_@89.105.194.91) has joined #ceph
[10:35] * CSa_ (~christian@mintzer.imp.fu-berlin.de) has joined #ceph
[10:40] * ChrisNBlum (~ChrisNBlu@dhcp-ip-230.dorf.rwth-aachen.de) has joined #ceph
[10:42] <theanalyst> I've a couple of osd's which are down in a cluster...restarting makes the cluster health OK.. but how can I find the cause to OSD's going down
[10:44] * kefu (~kefu@119.254.120.68) has joined #ceph
[10:47] * fmanana (~fdmanana@bl13-135-31.dsl.telepac.pt) has joined #ceph
[10:50] * joao (~joao@249.38.136.95.rev.vodafone.pt) has joined #ceph
[10:50] * ChanServ sets mode +o joao
[10:52] * zack_dolby (~textual@pa3b3a1.tokynt01.ap.so-net.ne.jp) Quit (Quit: My MacBook has gone to sleep. ZZZzzz???)
[10:53] * jashank42 (~jashan42@117.214.250.47) Quit (Ping timeout: 480 seconds)
[10:56] * calvinx (~calvin@101.100.172.246) has joined #ceph
[10:58] * bjornar_ (~bjornar@93.187.84.175) has joined #ceph
[10:59] * Miouge (~Miouge@94.136.92.20) has joined #ceph
[11:02] * jashank42 (~jashan42@117.220.137.174) has joined #ceph
[11:02] * Grimmer (~Chrissi_@5NZAADHXG.tor-irc.dnsbl.oftc.net) Quit ()
[11:03] * Lattyware (~Nephyrin@ncc-1701-a.tor-exit.network) has joined #ceph
[11:04] * oblu (~o@62.109.134.112) Quit (Remote host closed the connection)
[11:05] * daviddcc (~dcasier@84.197.151.77.rev.sfr.net) Quit (Ping timeout: 480 seconds)
[11:09] * squ (~Thunderbi@46.109.36.167) Quit (Ping timeout: 480 seconds)
[11:11] * ngoswami (~ngoswami@121.244.87.116) has joined #ceph
[11:23] * lucas1 (~Thunderbi@218.76.52.64) has joined #ceph
[11:25] * lucas1 (~Thunderbi@218.76.52.64) Quit ()
[11:26] * cooldharma06 (~chatzilla@14.139.180.40) Quit (Remote host closed the connection)
[11:31] * kefu is now known as kefu|afk
[11:31] * kefu|afk (~kefu@119.254.120.68) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[11:32] * Lattyware (~Nephyrin@8Q4AABD3F.tor-irc.dnsbl.oftc.net) Quit ()
[11:35] <tuxcraft1r> where do i find the default setitngs values of ceph for 0.80.9
[11:36] * Miouge (~Miouge@94.136.92.20) Quit (Quit: Miouge)
[11:36] <tuxcraft1r> like rbd cache = ? true ? false ?
[11:37] * shnarch (~shnarch@2.52.61.121) Quit (Ping timeout: 480 seconds)
[11:40] <theanalyst> ceph --admin-daemon /path/to/mon.asok config show
[11:41] * thomnico (~thomnico@2a01:e35:8b41:120:a996:3c09:b684:4b7a) Quit (Ping timeout: 480 seconds)
[11:42] * ira (~ira@c-71-233-225-22.hsd1.ma.comcast.net) has joined #ceph
[11:42] <theanalyst> tuxcraft1r: http://ceph.com/docs/master/rados/troubleshooting/log-and-debug/#runtime
[11:46] * xinxinsh (~xinxinsh@shzdmzpr02-ext.sh.intel.com) Quit (Ping timeout: 480 seconds)
[11:54] * thomnico (~thomnico@2a01:e35:8b41:120:a996:3c09:b684:4b7a) has joined #ceph
[11:57] * antoine (~bourgault@192.93.37.4) Quit (Ping timeout: 480 seconds)
[11:58] * thomnico (~thomnico@2a01:e35:8b41:120:a996:3c09:b684:4b7a) Quit ()
[12:01] * dugravot61 (~dugravot6@nat-persul-plg.wifi.univ-lorraine.fr) Quit (Ping timeout: 480 seconds)
[12:03] * pakman__ (~rushworld@destiny.enn.lu) has joined #ceph
[12:03] * cooldharma06 (~chatzilla@14.139.180.40) has joined #ceph
[12:04] * badone (~brad@CPE-121-215-241-179.static.qld.bigpond.net.au) has joined #ceph
[12:05] * jashank42 (~jashan42@117.220.137.174) Quit (Ping timeout: 480 seconds)
[12:05] * zack_dolby (~textual@pw126152073024.10.panda-world.ne.jp) has joined #ceph
[12:05] * dgurtner_ (~dgurtner@178.197.231.53) has joined #ceph
[12:05] * zack_dolby (~textual@pw126152073024.10.panda-world.ne.jp) Quit ()
[12:07] * dgurtner (~dgurtner@178.197.231.155) Quit (Ping timeout: 480 seconds)
[12:07] <tuxcraft1r> theanalyst: thx!
[12:10] * Jules (~oftc-webi@210.213.71.80) has joined #ceph
[12:10] <Jules> Good day to everyone, Have anyone here tried deploying ceph/ceph-docker?
[12:13] * dugravot6 (~dugravot6@dn-infra-04.lionnois.univ-lorraine.fr) has joined #ceph
[12:14] * thomnico (~thomnico@2a01:e35:8b41:120:a996:3c09:b684:4b7a) has joined #ceph
[12:16] * Jules (~oftc-webi@210.213.71.80) Quit (Quit: Page closed)
[12:18] * jashank42 (~jashan42@117.214.255.176) has joined #ceph
[12:20] * Jules (~oftc-webi@210.213.71.80) has joined #ceph
[12:25] * Concubidated (~Adium@66-87-127-8.pools.spcsdns.net) Quit (Quit: Leaving.)
[12:26] * badone (~brad@CPE-121-215-241-179.static.qld.bigpond.net.au) Quit (Ping timeout: 480 seconds)
[12:26] * xinxinsh (~xinxinsh@shzdmzpr02-ext.sh.intel.com) has joined #ceph
[12:31] * jashank42 (~jashan42@117.214.255.176) Quit (Ping timeout: 480 seconds)
[12:31] * cok (~chk@2a02:2350:18:1010:e826:21c4:abff:168) has left #ceph
[12:33] * pakman__ (~rushworld@5NZAADH00.tor-irc.dnsbl.oftc.net) Quit ()
[12:33] * Frostshifter (~CobraKhan@ncc-1701-d.tor-exit.network) has joined #ceph
[12:34] * Jules (~oftc-webi@210.213.71.80) Quit (Quit: Page closed)
[12:35] * spow (~spow@95.45.132.206) Quit (Remote host closed the connection)
[12:35] * spow (~spow@95.45.132.206) has joined #ceph
[12:35] * oblu (~o@62.109.134.112) has joined #ceph
[12:36] * spow (~spow@95.45.132.206) Quit (Remote host closed the connection)
[12:36] <tuxcraft1r> how do you distrubute the unique secret key used for libvirt to access the rbd when using virsh secret-define --file secret.xml
[12:37] <tuxcraft1r> because it is stopping me from being able to migrate the vm guest because my kvm host are having unique ids to access ceph
[12:37] * lucas1 (~Thunderbi@218.76.52.64) has joined #ceph
[12:41] * jashank42 (~jashan42@117.215.150.68) has joined #ceph
[12:41] * nsoffer (~nsoffer@bzq-79-180-80-86.red.bezeqint.net) has joined #ceph
[12:45] * dgurtner_ (~dgurtner@178.197.231.53) Quit (Ping timeout: 480 seconds)
[12:51] * zack_dolby (~textual@pw126152073024.10.panda-world.ne.jp) has joined #ceph
[12:52] * ade (~abradshaw@tmo-100-250.customers.d1-online.com) Quit (Ping timeout: 480 seconds)
[12:54] * dgurtner (~dgurtner@178.197.231.53) has joined #ceph
[12:56] * bene (~ben@c-24-60-237-191.hsd1.nh.comcast.net) has joined #ceph
[12:57] * ade (~abradshaw@tmo-101-101.customers.d1-online.com) has joined #ceph
[12:58] * lucas1 (~Thunderbi@218.76.52.64) Quit (Quit: lucas1)
[13:01] * lucas1 (~Thunderbi@218.76.52.64) has joined #ceph
[13:02] * shang (~ShangWu@175.41.48.77) Quit (Remote host closed the connection)
[13:03] * Frostshifter (~CobraKhan@7R2AABKX8.tor-irc.dnsbl.oftc.net) Quit ()
[13:03] * Nanobot (~cheese^@static-ip-85-25-103-119.inaddr.ip-pool.com) has joined #ceph
[13:04] * fmanana (~fdmanana@bl13-135-31.dsl.telepac.pt) Quit (Ping timeout: 480 seconds)
[13:04] * sankarshan (~sankarsha@183.87.39.242) has joined #ceph
[13:09] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit (Ping timeout: 480 seconds)
[13:09] * calvinx (~calvin@101.100.172.246) Quit (Quit: calvinx)
[13:14] * jashank42 (~jashan42@117.215.150.68) Quit (Ping timeout: 480 seconds)
[13:16] * nsoffer (~nsoffer@bzq-79-180-80-86.red.bezeqint.net) Quit (Ping timeout: 480 seconds)
[13:21] * lucas1 (~Thunderbi@218.76.52.64) Quit (Quit: lucas1)
[13:23] * jashank42 (~jashan42@117.220.142.77) has joined #ceph
[13:25] * daviddcc (~dcasier@LCaen-656-1-144-187.w217-128.abo.wanadoo.fr) has joined #ceph
[13:25] * bjornar_ (~bjornar@93.187.84.175) Quit (Remote host closed the connection)
[13:25] * treenerd (~treenerd@85.193.140.98) has joined #ceph
[13:25] * fmanana (~fdmanana@bl13-135-31.dsl.telepac.pt) has joined #ceph
[13:26] * m0zes (~mozes@beocat.cis.ksu.edu) Quit (Ping timeout: 480 seconds)
[13:27] * karnan (~karnan@121.244.87.117) Quit (Remote host closed the connection)
[13:28] * vbellur (~vijay@121.244.87.117) Quit (Ping timeout: 480 seconds)
[13:28] * flisky1 (~Thunderbi@106.39.60.34) has joined #ceph
[13:31] * flisky (~Thunderbi@106.39.60.34) Quit (Ping timeout: 480 seconds)
[13:31] * ChrisNBlum (~ChrisNBlu@dhcp-ip-230.dorf.rwth-aachen.de) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[13:33] * Nanobot (~cheese^@3DDAAAXX2.tor-irc.dnsbl.oftc.net) Quit ()
[13:33] * Da_Pineapple (~Nanobot@chell.quenelle.pw) has joined #ceph
[13:33] * sleinen1 (~Adium@2001:620:0:2d:7ed1:c3ff:fedc:3223) has joined #ceph
[13:33] * wschulze (~wschulze@cpe-69-206-240-164.nyc.res.rr.com) has joined #ceph
[13:34] * flisky (~Thunderbi@106.39.60.34) has joined #ceph
[13:34] * davidz1 (~davidz@cpe-23-242-27-128.socal.res.rr.com) has joined #ceph
[13:35] * flisky (~Thunderbi@106.39.60.34) Quit ()
[13:37] * rotbeard (~redbeard@x5f74c14d.dyn.telefonica.de) has joined #ceph
[13:37] * overclk (~overclk@121.244.87.117) Quit (Quit: Leaving)
[13:38] * davidz (~davidz@2605:e000:1313:8003:f1d6:5146:cf14:a74b) Quit (Read error: Connection reset by peer)
[13:38] * flisky1 (~Thunderbi@106.39.60.34) Quit (Ping timeout: 480 seconds)
[13:39] * zhaochao (~zhaochao@125.39.8.226) Quit (Quit: ChatZilla 0.9.91.1 [Iceweasel 38.0.1/20150526223604])
[13:40] * sleinen (~Adium@macsl.switch.ch) Quit (Ping timeout: 480 seconds)
[13:41] * vbellur (~vijay@121.244.87.124) has joined #ceph
[13:44] * willemb (~willemb@vpn-router1.dur1.host-h.net) has joined #ceph
[13:44] * fmanana (~fdmanana@bl13-135-31.dsl.telepac.pt) Quit (Ping timeout: 480 seconds)
[13:46] <willemb> Hi. I am trying to find out why the model this guy describes is so popular: http://www.sebastien-han.fr/blog/2012/07/06/nfs-over-rbd/ why not just map and mount the block devices directly on the clients?
[13:47] <Be-El> willemb: because rbd is not available on all clients
[13:47] * jashank42 (~jashan42@117.220.142.77) Quit (Ping timeout: 480 seconds)
[13:47] <Be-El> willemb: and you do not want to expose the ceph network to all clients in some cases
[13:48] <willemb> https://ceph.com/user-story/ceph-from-poc-to-production/
[13:48] <willemb> I believe that was written by the same guy. Possibly about the same implementation
[13:48] <willemb> And I think the use-case at stone-it is similar to what we have in mind
[13:49] <rotbeard> wicope, especially for vmware we need that kind of proxy system
[13:50] <Be-El> willemb: you may also want to have a look at cephfs and ganesha nfs
[13:50] <willemb> i've been ignoring cephfs because everyone says it is not production ready
[13:51] * jclm1 (~jclm@121.244.87.117) Quit (Quit: Leaving.)
[13:51] <Be-El> willemb: some features are not ready for prime time, yes...like active/active mds or snapshots
[13:52] * shylesh (~shylesh@121.244.87.124) Quit (Remote host closed the connection)
[13:52] <willemb> ok, so if my plan is to create some 500 000 block devices, one for each site we host, and mount them into /var/www/siteX . Would you reccommend doing that by using the ceph client on jessie, or having an nfs tier in-between?
[13:53] <Be-El> 500000 mount points?
[13:54] <willemb> obviously i have to break that up per server somehow
[13:54] <Be-El> if you can afford to skip nfs you should probably skip it
[13:54] <Be-El> do you use virtual machines for hosting?
[13:55] <Be-El> with a standard nfs server you will either have a single point of failure or a complex setup for HA
[13:55] <Be-El> that's why I mentioned ganesha nfs
[13:55] <willemb> I'll look into it
[13:55] <willemb> We don't currently
[13:56] <Be-El> the main problem with block devices is the fact that only one client should access them at any time
[13:56] <willemb> and we limit tenancy to about 600 sites per server, due to how many customers can be impacted by an outage
[13:56] <willemb> I am considering cutting this up into 200 site chunks and putting them on vm's
[13:56] <Be-El> if you have to move a site to a different host e.g. in case of a server crash, you'll be in trouble
[13:56] <willemb> it will mean double as many public ips and double the license fee for things like ksplice/uptrack
[13:56] <rotbeard> Be-El, could you point out the advantages of ganesha nfs over the _classical_ nfs kernel server setting inside a pacemaker cluster (sorry for offtopic)
[13:57] * jashank42 (~jashan42@59.89.62.76) has joined #ceph
[13:57] <Be-El> rotbeard: i haven't used ganesha yet, but it has native support for cephfs and support for pNFS (multiple servers)
[13:57] <willemb> Be-El: i'll be in less trouble than what I am in when that happens now, with the disks in the box
[13:57] <rotbeard> ah ok. that sounds pretty nice ;) (if it is enterprise ready and stuff)
[13:58] <Be-El> https://github.com/nfs-ganesha/nfs-ganesha/wiki
[13:58] <Be-El> rotbeard: no clue about its current state
[13:59] <Be-El> rotbeard: i'm thinking about replacing our nfs box with a ganesha host and cephfs, that why I had a look at it
[13:59] <rotbeard> I should really have a look on that, thanks Be-El
[13:59] <willemb> why do you say i will be in trouble if i have to move a site to a different host?
[13:59] <willemb> just boot, do a puppet run to get the config right, and that will mount the right nfs share, boom done?
[14:01] * ngoswami (~ngoswami@121.244.87.116) Quit (Quit: Leaving)
[14:01] <rotbeard> Be-El, I fully understand. we had a lot of trouble last year with our NFS proxy systems... mostly our own fault, because some folks put my testing clusters into production while I was in holiday ;) with NFS + HA you really have to be careful about things like fencing, ceph client limits et cetera
[14:01] * dneary (~dneary@nat-pool-bos-u.redhat.com) has joined #ceph
[14:02] <Be-El> willemb: if you use rbd and virtual machines with internal page cache, in case of a crash the page cache will not be flushed and the filesystem on the rbd may be inconsistent
[14:02] <willemb> even if i don't put the root filesystem on rbd?
[14:03] <willemb> how would that be different on a virtual machine than on metal|?
[14:03] * Da_Pineapple (~Nanobot@9S0AAAV61.tor-irc.dnsbl.oftc.net) Quit ()
[14:04] * KevinPerks (~Adium@2606:a000:80ad:1300:6584:ea02:b68e:1898) has joined #ceph
[14:04] <Be-El> willemb: it won't be different
[14:04] <willemb> ok, and if i put nfs or ganesha nfs in-between?
[14:04] * i_m (~ivan.miro@deibp9eh1--blueice1n2.emea.ibm.com) has joined #ceph
[14:05] <Be-El> willemb: with rbd you have to ensure that the device are properly unmapped and not accessed from multiple clients at the same time
[14:05] <willemb> thanks. does that require special effort?
[14:05] <Be-El> willemb: for nfs the nfs server has to guarantee the consistency of files. if several nfs clients access the same file, the typical nfs file access semantics kick in
[14:05] * ngoswami (~ngoswami@121.244.87.116) has joined #ceph
[14:06] * vbellur (~vijay@121.244.87.124) Quit (Ping timeout: 480 seconds)
[14:06] <willemb> i was planning on naming the rbds after what i will use them for, and only mount them on one machine at a time on purpose
[14:07] <T1w> a quick clearing-up question.. Will ganesha give me anything if I just need a NFS-front for a couple (3-5) RBD volumes and I can live with having to perform a manual failover to a new NFS-front?
[14:08] <Be-El> willemb: the problematic part the "on purpose". consider a temporary network failure that disconnects one host with mapped rbds. you start the site on a different host to make it available again. five minutes later the network problem is resolved, and the former clients continues to access the rbd at the state it left it. BAM
[14:09] <Be-El> T1w: as far as I understand ganesha it just acts as an nfs server. but it has a configurable abstraction layer for file systems exported via nfs. one implementation of this layer is based on cepfs
[14:09] <T1w> Be-El: ok
[14:09] <Be-El> T1w: so in case of rbd you will gain nothing (but the usual HA/fencing trouble)
[14:09] <willemb> ok, so putting an nfs server in-between will limit the number of machines that can potentially cause havoc
[14:09] <T1w> seems like I'm better of just handling that myself then
[14:10] <Be-El> willemb: the client just talk to the nfs server. the server has to handle possible conflicts
[14:11] <Be-El> willemb: how this is done is defined for nfs
[14:11] <willemb> yes, but the nfs server could also temporarily dissappear from the network?
[14:11] <Be-El> sure
[14:11] <Be-El> .oO ( i should definitely have a closer look at ganesha and pnfs... )
[14:13] * flisky (~Thunderbi@106.39.60.34) has joined #ceph
[14:13] <T1w> I assume that a RBD can be resized and then a filesystem inside can be grown without problems?
[14:13] * georgem (~Adium@fwnat.oicr.on.ca) has joined #ceph
[14:13] <Be-El> willemb: with a standard nfs server you will have a single point of failure
[14:13] <T1w> is. a RBD with an ext4 fs in it
[14:13] <T1w> ie even
[14:13] <Be-El> T1w: yes, qemu-img will to the jobs for resizing the rbd
[14:14] <Be-El> T1w: whether the filesystem can be resized online depends on the filesystem implementation
[14:14] <T1w> resize2fs or lvm qould be my choice
[14:14] <T1w> qould even
[14:15] <T1w> actually I'm considering adding additional space via LVM - add a new physical device to the same volumegroup and expand
[14:15] <T1w> the worst limit of doing that would be inode allocation at fs creation
[14:15] * jdillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) has joined #ceph
[14:16] <Be-El> T1w: i think it's not worth the trouble and complexity. you can use lvm on the rbd, and then resize the rbd and use pvresize
[14:16] <circ-user-ClZ7Y> following discussion above, seems rbd is in no state to use as multi-mount point. Wanna use cephfs, and using it on test-env already, but it seems that if i mount it in "kernel-mode" clientA ls freeze while clientB is heavily writing multiple files. Fuse is better, but still not much
[14:16] * circ-user-ClZ7Y is now known as emik0
[14:17] <T1w> Be-El: now that you mention it, yes.. adding a new volume is redundant since I can just grow the rbd
[14:17] <emik0> any best-practise advices?
[14:17] <Be-El> emik0: rbd is just a block device, no magic included. you cannot access it from multiple clients like you cannot attach a spinning disk to multiple hosts
[14:17] <Be-El> _except_ if you use a filesystem that is aware of it like OCFS2
[14:17] <T1w> it would certanly also limit the amount of rbd mounts needed
[14:17] <emik0> yeah, thats why i'm asking about cephFS
[14:18] <T1w> afk
[14:18] <Be-El> emik0: which version of ceph do you use?
[14:18] <janos_> emik0, you said "seems rbd is in no state to use as multi-mount point" - that implies it's deficient but will be "fixed"
[14:18] <perpetualrabbit> request for help: I'm trying to set up a ceph cluster manually, without ceph deploy.
[14:19] <perpetualrabbit> I set up one mon, now I try to set up one osd.
[14:19] <perpetualrabbit> That seems to go wrong at the beginning:
[14:19] <perpetualrabbit> uuidgen
[14:19] <perpetualrabbit> output: 147f66f6-44cb-4c73-b0de-74e630653848
[14:19] <emik0> hammer, latest from git, debian-packaged
[14:19] <perpetualrabbit> Then: ceph osd create 147f66f6-44cb-4c73-b0de-74e630653848
[14:20] * flisky (~Thunderbi@106.39.60.34) Quit (Quit: flisky)
[14:20] <emik0> sorry, said it wrong, using cephfs only, no intentions with rbd
[14:20] <perpetualrabbit> output: Error initializing cluster client: Error('error calling conf_read_file: errno EINVAL',)
[14:20] <willemb> Be-El: Do you reccommend a rbd per server or a rbd per site that is supposed to be on the server?
[14:20] <Be-El> emik0: same here, but latest ubuntu release. cephfs works fine for me with about 20 clients, 5 mons, 7 osds and one mds
[14:20] <perpetualrabbit> this is on the osd node, so what am I doing wrong? What does the error mean?
[14:21] <Be-El> perpetualrabbit: there's no configuration file?
[14:21] * jashank42 (~jashan42@59.89.62.76) Quit (Ping timeout: 480 seconds)
[14:21] <emik0> I've tested only 3-node configuration. 2 osd, 3 mon, 1 mds. Latest ubuntu = what version ceph?
[14:21] <Be-El> willemb: i cannot recommend a solution
[14:21] <perpetualrabbit> Be-El, I guess so, but should there be one? If so, what should be in it?
[14:21] <Be-El> emik0: hammer, too
[14:22] <Be-El> perpetualrabbit: to deploy an osd you need at least information about the mons and the admin key
[14:22] <Be-El> perpetualrabbit: and fsid, networks etc.
[14:22] <emik0> do you mount it via fuse? afk
[14:22] <Be-El> emik0: yes
[14:23] <perpetualrabbit> Do I need to copy the ceph.conf file on the mon to all my osd's?
[14:23] <perpetualrabbit> The manual does not mention it
[14:23] <Be-El> perpetualrabbit: i'm managing the nodes with puppet and a heavily modified puppet-ceph module
[14:24] <Be-El> it takes care for setting up everything
[14:24] * jclm (~jclm@122.181.21.134) has joined #ceph
[14:24] <Be-El> (ok, haven't tested a complete bootstrap for months... )
[14:24] <perpetualrabbit> Be-El, that sounds really nice
[14:26] * kanagaraj (~kanagaraj@121.244.87.117) Quit (Quit: Leaving)
[14:27] <emik0> BTW can anyone help me out with understanding of OSD size? it seems docs are uninformative, can't find any info how much space is used
[14:27] <perpetualrabbit> Be-El, but I don't know puppet and that would make two unknown systems for me instead of just one. But for future deployments thanks for the tip!
[14:27] * fmanana (~fdmanana@bl13-135-31.dsl.telepac.pt) has joined #ceph
[14:30] <perpetualrabbit> Be-El, ok, I got a further, although I don't know what I'm doing. I copied /etc/ceph.conf and /etc/ceph/ceph.client.admin.keyring to the osd's /etc/ceph directory. Now doing ceph osd create 147f66f6-44cb-4c73-b0de-74e630653848 gives the output: 0
[14:30] <Be-El> perpetualrabbit: so it created an osd with id 0
[14:31] <Be-El> perpetualrabbit: does ceph -s give you output?
[14:31] <perpetualrabbit> so now I suppose I must start it. Back to the docs
[14:31] <Be-El> perpetualrabbit: nope
[14:31] <Be-El> perpetualrabbit: you only allocated an osd id
[14:32] <Be-El> perpetualrabbit: i would recommend to have a look at "ceph-disk prepare" and "ceph-disk activate"
[14:32] <Be-El> and you do not need to allocate id by yourself
[14:33] * Kealper (~JamesHarr@nx-74205.tor-exit.network) has joined #ceph
[14:33] * georgem (~Adium@fwnat.oicr.on.ca) Quit (Quit: Leaving.)
[14:33] * zack_dolby (~textual@pw126152073024.10.panda-world.ne.jp) Quit (Quit: My MacBook has gone to sleep. ZZZzzz???)
[14:36] * Infected (infected@185.18.76.73) Quit (Quit: leaving)
[14:41] * lucas1 (~Thunderbi@218.76.52.64) has joined #ceph
[14:43] * arbrandes (~arbrandes@177.138.114.132) has joined #ceph
[14:43] * linuxkidd (~linuxkidd@207.236.250.131) has joined #ceph
[14:44] * debian112 (~bcolbert@24.126.201.64) has joined #ceph
[14:45] * xinxinsh (~xinxinsh@shzdmzpr02-ext.sh.intel.com) Quit (Ping timeout: 480 seconds)
[14:45] * lucas1 (~Thunderbi@218.76.52.64) Quit ()
[14:46] <T1w> Be-El: is there an upper limit for rbd sizes?
[14:46] <Be-El> T1w: eh....i've no clue
[14:46] <T1w> ok
[14:47] <Be-El> T1w: but i've seen several people trying desparately to remove rbd with several PB size after they have mixed up parameters on resize.... ;-)
[14:47] <T1w> ahahahaha
[14:47] <T1w> I
[14:48] <T1w> I've seen enough use of lvextend that I know the importance of the small +
[14:48] <Be-El> even if ceph does not allocate all the memory upon resize, during delete it has to check all chunks and delete them separately
[14:49] <Be-El> the problem is not lvextend, but the qemu-img or rbd parameters...one of them is in kb instead of bytes
[14:49] <T1w> .. and I've also come to the realisation that I prefer a grahipcal interface to resizing - simply because its more initutive and I limit myself from doing domething disasterous
[14:49] <T1w> something even
[14:49] * dgurtner_ (~dgurtner@178.197.236.107) has joined #ceph
[14:51] * dgurtner (~dgurtner@178.197.231.53) Quit (Ping timeout: 480 seconds)
[14:54] * brajesh (~oftc-webi@103.27.8.44) has joined #ceph
[14:54] <brajesh> Hi
[14:54] <brajesh> how to clone vm when using ceph
[14:55] <brajesh> image created with qemu-img doesn't support layering
[14:55] * georgem (~Adium@fwnat.oicr.on.ca) has joined #ceph
[14:55] * TMM (~hp@sams-office-nat.tomtomgroup.com) Quit (Quit: Ex-Chat)
[14:59] * bene (~ben@c-24-60-237-191.hsd1.nh.comcast.net) Quit (Quit: Konversation terminated!)
[15:00] * mhack (~mhack@nat-pool-bos-t.redhat.com) Quit (Quit: Leaving)
[15:01] * sleinen1 (~Adium@2001:620:0:2d:7ed1:c3ff:fedc:3223) Quit (Quit: Leaving.)
[15:02] * TMM (~hp@sams-office-nat.tomtomgroup.com) has joined #ceph
[15:02] * bene (~ben@2601:18c:8300:2e88:56ee:75ff:fe10:7266) has joined #ceph
[15:03] * Kealper (~JamesHarr@3OZAAB5D6.tor-irc.dnsbl.oftc.net) Quit ()
[15:03] * Jyron (~Keiya@176.10.99.209) has joined #ceph
[15:06] * vbellur (~vijay@122.172.46.139) has joined #ceph
[15:06] * rlrevell (~leer@vbo1.inmotionhosting.com) has joined #ceph
[15:08] * alram (~alram@50.95.199.16) has joined #ceph
[15:09] * dephcon (~oftc-webi@c73-110.rim.net) has joined #ceph
[15:10] * brajesh (~oftc-webi@103.27.8.44) Quit (Quit: Page closed)
[15:13] * TMM (~hp@sams-office-nat.tomtomgroup.com) Quit (Ping timeout: 480 seconds)
[15:16] * dyasny (~dyasny@173.231.115.59) has joined #ceph
[15:17] * daviddcc (~dcasier@LCaen-656-1-144-187.w217-128.abo.wanadoo.fr) Quit (Ping timeout: 480 seconds)
[15:18] * sjm (~sjm@pool-173-70-76-86.nwrknj.fios.verizon.net) has joined #ceph
[15:19] * harold (~hamiller@71-94-227-123.dhcp.mdfd.or.charter.com) has joined #ceph
[15:20] * tupper_ (~tcole@173.38.117.67) has joined #ceph
[15:27] * sleinen (~Adium@130.59.94.206) has joined #ceph
[15:29] * sleinen1 (~Adium@2001:620:0:82::102) has joined #ceph
[15:30] * brad_mssw (~brad@66.129.88.50) has joined #ceph
[15:33] * Jyron (~Keiya@0SGAABANA.tor-irc.dnsbl.oftc.net) Quit ()
[15:34] * dugravot6 (~dugravot6@dn-infra-04.lionnois.univ-lorraine.fr) Quit (Quit: Leaving.)
[15:34] <willemb> is there some maximum number of rbd 's that I can map and mount on a single client?
[15:35] * sleinen (~Adium@130.59.94.206) Quit (Ping timeout: 480 seconds)
[15:38] * t0rn (~ssullivan@2607:fad0:32:a02:56ee:75ff:fe48:3bd3) has joined #ceph
[15:41] * jskinner (~jskinner@host-95-2-129.infobunker.com) has joined #ceph
[15:41] * t0rn (~ssullivan@2607:fad0:32:a02:56ee:75ff:fe48:3bd3) has left #ceph
[15:44] * kawa2014 (~kawa@89.184.114.246) Quit (Ping timeout: 480 seconds)
[15:44] <willemb> ie, is this still relevant?
[15:44] <willemb> http://www.spinics.net/lists/ceph-devel/msg09363.html
[15:45] * kawa2014 (~kawa@2001:67c:1560:8007::aac:c1a6) has joined #ceph
[15:51] * diegows (~diegows@190.190.5.238) has joined #ceph
[15:56] * marrusl (~mark@nat-pool-rdu-u.redhat.com) has joined #ceph
[15:57] <Be-El> willemb: each rbd is a block device. and there's an upper limit on block devices in a system
[15:57] * SkyEye (~gaveen@175.157.54.12) Quit (Remote host closed the connection)
[15:58] * cok (~chk@2a02:2350:18:1010:d48b:43fa:c16f:29af) has joined #ceph
[15:59] <willemb> there is even a limit on the number of block devices of the same type you can mount. and that seems to max out at either 200-ish or 800-ish :(
[16:00] <willemb> or am I confusing two things here?
[16:00] * DV_ (~veillard@2001:41d0:a:f29f::1) has joined #ceph
[16:03] * rwheeler (~rwheeler@nat-pool-bos-t.redhat.com) has joined #ceph
[16:03] * Quackie (~Snowman@ncc-1701-d.tor-exit.network) has joined #ceph
[16:04] <Be-El> what are the major and minor numbers used for rbd devices?
[16:04] <doppelgrau> willemb: what are you planing to do? If you can do it in the userspace there won???t be an limit
[16:06] * T1w (~jens@node3.survey-it.dk) Quit (Ping timeout: 480 seconds)
[16:07] * alram (~alram@50.95.199.16) Quit (Quit: leaving)
[16:07] * Hemanth (~Hemanth@121.244.87.117) Quit (Ping timeout: 480 seconds)
[16:08] * DV (~veillard@2001:41d0:1:d478::1) Quit (Ping timeout: 480 seconds)
[16:08] * ChrisNBlum (~ChrisNBlu@dhcp-ip-230.dorf.rwth-aachen.de) has joined #ceph
[16:08] * amote (~amote@121.244.87.116) Quit (Quit: Leaving)
[16:11] * cok (~chk@2a02:2350:18:1010:d48b:43fa:c16f:29af) Quit (Quit: Leaving.)
[16:11] <perpetualrabbit> Ok, so I got an osd configured, but it will not start. Cause: the mon has two ip addresses. The hostname of the mon is the dns name of the outer network, and the internal 10.0.0.0/16 net is where both the osd as the mon's are on. The error I see on the osd is this: ceph-osd -i 0
[16:11] <perpetualrabbit> warning: line 12: 'filestore_xattr_use_omap' in section 'global' redefined
[16:11] <perpetualrabbit> 2015-06-09 15:28:55.416336 7f44884e6840 -1 unable to find any IP address in networks: 132.229.226.0/28
[16:12] <perpetualrabbit> What do I do now?
[16:14] <Be-El> configure the correct network interfaces for the osd host
[16:14] * jcalcote_ (~oftc-webi@63-248-159-172.static.layl0102.digis.net) Quit (Remote host closed the connection)
[16:21] * yanzheng (~zhyan@171.216.94.129) Quit (Quit: This computer has gone to sleep)
[16:26] <perpetualrabbit> Be-El, of course network is up and running for the osd, so I had to try something else. I removed the 'public network' and 'cluster network' lines from /etc/ceph/ceph.conf on the osd and now it works. Does that make sense?
[16:28] <Be-El> sure, since these settings define the networks for public and private traffic. if your host does not define a network interface within these networks, the osd cannot start
[16:29] <perpetualrabbit> which host do you mean, the mon or the osd?
[16:31] <perpetualrabbit> Initially I defined public network = 132.229.226.0/28 and cluster network = 10.0.0.0/16 on both the mon and the osd. Since the osd has only the 10.0.0.0 network, I assumed it would pick the right one.
[16:32] * Cybert1nus (~Cybertinu@cybertinus.customer.cloud.nl) has joined #ceph
[16:32] * Concubidated (~Adium@192.41.52.12) has joined #ceph
[16:33] * Quackie (~Snowman@5NZAADIAY.tor-irc.dnsbl.oftc.net) Quit ()
[16:33] * Corti^carte (~toast@7R2AABK34.tor-irc.dnsbl.oftc.net) has joined #ceph
[16:33] * peem_ (~oftc-webi@office.forlinux.co.uk) has joined #ceph
[16:34] <peem_> anybody had experience with radosgw and inkscope in particular ? I need some help setting it up.
[16:35] * rotbeard (~redbeard@x5f74c14d.dyn.telefonica.de) Quit (Quit: Leaving)
[16:36] * oro (~oro@2001:620:20:16:159d:8f9e:2ff5:5914) has joined #ceph
[16:36] * Cybertinus (~Cybertinu@cybertinus.customer.cloud.nl) Quit (Ping timeout: 480 seconds)
[16:39] * sleinen1 (~Adium@2001:620:0:82::102) Quit (Read error: Connection reset by peer)
[16:42] <Be-El> perpetualrabbit: public and private network serve two different purposes
[16:43] * georgem1 (~Adium@fwnat.oicr.on.ca) has joined #ceph
[16:46] * Hau_MI (~HauM1@login.univie.ac.at) has joined #ceph
[16:47] * morse (~morse@supercomputing.univpm.it) has joined #ceph
[16:47] * philips (~philips@ec2-54-196-103-51.compute-1.amazonaws.com) Quit (Ping timeout: 480 seconds)
[16:48] * marrusl (~mark@nat-pool-rdu-u.redhat.com) Quit (Ping timeout: 480 seconds)
[16:48] * georgem (~Adium@fwnat.oicr.on.ca) Quit (Ping timeout: 480 seconds)
[16:48] * HauM1 (~HauM1@login.univie.ac.at) Quit (Ping timeout: 480 seconds)
[16:48] * morse_ (~morse@supercomputing.univpm.it) Quit (Ping timeout: 480 seconds)
[16:49] * marrusl (~mark@nat-pool-rdu-u.redhat.com) has joined #ceph
[16:49] * xarses (~andreww@50.141.6.2) Quit (Remote host closed the connection)
[16:51] <perpetualrabbit> Be-El, Yes I understand that. cluster network is for internal communication between mon's and osd's, and among osd's. Public network is for users of the cluster.
[16:51] <Be-El> perpetualrabbit: nope
[16:52] <Be-El> perpetualrabbit: private network is for backfilling and heartbeat between osd. mons do not use the private network
[16:52] * philips (~philips@ec2-54-196-103-51.compute-1.amazonaws.com) has joined #ceph
[16:52] <Be-El> perpetualrabbit: public network is for clients, mons and osds
[16:52] * yanzheng (~zhyan@171.216.94.129) has joined #ceph
[16:53] <perpetualrabbit> Ok, so I have a cluster. three hosts have both internal (10.0.0.0/16) network and outside (132.229.226.0/28). All others (70 nodes) only have the 10.0.0.0/16 net.
[16:54] <perpetualrabbit> What should I put in ceph.conf on the mons, and in the osds?
[16:54] <Be-El> perpetualrabbit: and that's wrong
[16:54] * jwilkins (~jwilkins@2601:9:4580:f4c:ea2a:eaff:fe08:3f1d) has joined #ceph
[16:54] * vikhyat (~vumrao@121.244.87.116) Quit (Quit: Leaving)
[16:55] <perpetualrabbit> Why is that wrong?
[16:55] <perpetualrabbit> More importantly, what should it be then?
[16:57] <Be-El> perpetualrabbit: as i said above....public network is used by clients, mons and osds. so every host that acts as client, mon or osd must have access to this network. for clients routable access is ok, mon and osd need an interface in that network
[16:57] <Be-El> perpetualrabbit: and the private networks is for osds only
[16:58] <perpetualrabbit> ok, so I must set public network = 10.0.0.0/16 then. Do I need the private network at all? BTW private network is cluster network in the ceph.conf file, right?
[16:58] * alram (~alram@192.41.52.12) has joined #ceph
[16:59] <Be-El> perpetualrabbit: no
[16:59] <Be-El> perpetualrabbit: first of all you need to get back to your design and _define_ which network has which role
[17:00] <Be-El> perpetualrabbit: then you should take care for the host configuration and network accessibility
[17:00] * xarses (~andreww@50.141.6.2) has joined #ceph
[17:00] * kefu (~kefu@60.247.111.66) has joined #ceph
[17:01] * harold (~hamiller@71-94-227-123.dhcp.mdfd.or.charter.com) Quit (Quit: Leaving)
[17:03] * Corti^carte (~toast@7R2AABK34.tor-irc.dnsbl.oftc.net) Quit ()
[17:03] * sleinen (~Adium@2001:620:0:2d:7ed1:c3ff:fedc:3223) has joined #ceph
[17:03] * dicko (~arsenaali@ncc-1701-a.tor-exit.network) has joined #ceph
[17:05] <perpetualrabbit> Be-El, right. Maybe you can advise me? I have a 73 node cluster. The headnode is also a NAT gateway. That has the 10.0 and the 132.229.226 networks. All the nodes have only 10.0 addresses. The scientist's workstations are not in this 10.0 net. The headnode does NAT for the computing nodes to protect them from outside. In this situation I want to set up ceph. I read the ceph documentation but I still don't know if it is possible
[17:05] <perpetualrabbit> in this situation.
[17:06] <rlrevell> peem_: i've just set up radosgw. i recommend using hammer, it's way easier to set up
[17:06] <Be-El> perpetualrabbit: all nodes/workstations that should have access to ceph need to have a routable access to the osd hosts
[17:07] <Be-El> perpetualrabbit: so if the workstations are not in the 10.0.0.0/8 network and there's not tcp route to this network, they cannot access ceph
[17:07] <perpetualrabbit> ah I see
[17:08] <Be-El> perpetualrabbit: if the headnode acts as a NAT router from the workstations to the osd, you'll have a routable path. but that setup is useless, since you discard almost all advantages of ceph and get another single point of failure
[17:08] <perpetualrabbit> that sucks, because routable access is not going to happen there. So cephfs is out of the question for the workstations then. However, for the computing nodes, which will double as osd's, it will work.
[17:08] * b0e (~aledermue@213.95.25.82) Quit (Quit: Leaving.)
[17:09] <Be-El> perpetualrabbit: right
[17:09] * Nacer (~Nacer@252-87-190-213.intermediasud.com) Quit (Quit: Leaving...)
[17:09] <Be-El> perpetualrabbit: but keep in mind that osds also need compute capacities. many people do not mix osd and compute on the same host
[17:10] <perpetualrabbit> Also I could make block devices and export those via NFS. And run virtual machines based on those block devices in the cluster,
[17:10] <Be-El> perpetualrabbit: we had a discussion about cephfs and nfs earlier this day on the channel. maybe you have it in your backlog
[17:11] <perpetualrabbit> These are pretty strong machines. 128GiB of RAM, and 48-64 cores per machine
[17:11] * sleinen (~Adium@2001:620:0:2d:7ed1:c3ff:fedc:3223) Quit (Quit: Leaving.)
[17:11] <Be-El> perpetualrabbit: for vms inside the cluster you to not need nfs if the vm can access rbd images directly (e.g. kvm, qemu)
[17:11] <perpetualrabbit> That should be allright with that power.
[17:12] <Be-El> perpetualrabbit: and if a single compute jobs blocks the machine completely the osd will be dropped from the cluster since it is not able to respond to requests or heartbeats
[17:12] <perpetualrabbit> Yes, vms based on rbd images, that was the plan (one of the plans)
[17:13] <perpetualrabbit> will an osd come back automatically?
[17:13] * sleinen (~Adium@2001:620:0:2d:7ed1:c3ff:fedc:3223) has joined #ceph
[17:13] <Be-El> you can bind the osd daemons to 1-2 cores and keep the vms from blocking these cores, but I/O will still be contended
[17:14] <Be-El> perpetualrabbit: sure, but if it takes to long the cluster will start to backfill the data, resulting in laggy access to the data, more traffic, more load on osd, more osds being kicked out etc.
[17:14] <perpetualrabbit> We mostly have heavy computation and heavy memory usage, not so much I/O.
[17:14] * branto (~branto@ip-213-220-214-203.net.upcbroadband.cz) has left #ceph
[17:15] <Be-El> I/O also refers to cpu<->memory and cpu<->pci ;-)
[17:15] <perpetualrabbit> backfilling is making copies or more erasure coding, right?
[17:16] <Be-El> backfilling ensures that enough replicates / ec chunks are available in the cluster, yes
[17:16] <perpetualrabbit> yes, I see about I/O
[17:16] <Be-El> if one osd is down, its content needs to be replicated again to ensure consistency
[17:17] <Be-El> the extra network ("private network") is solely used for backfilling and heartbeats. using a distinct networks allows for better control about network contention, accessibility for clients etc.
[17:17] <perpetualrabbit> right. I need an estimate how much cpu and IO would need to be reserved for say 1 TB of storage per node. Also I need to know how I can reserve those resources.
[17:18] <Be-El> you should also reserve some memory. during backfilling the osd daemons grow in size
[17:19] * harold (~hamiller@71-94-227-123.dhcp.mdfd.or.charter.com) has joined #ceph
[17:19] <Be-El> and each kbyte of free memory can be used as cache for the osd's filesystem
[17:19] <perpetualrabbit> ok, but that is normal. Linux always does use as much memory for fs cache as it can
[17:20] * harold (~hamiller@71-94-227-123.dhcp.mdfd.or.charter.com) Quit ()
[17:20] <Be-El> sure. but if memory is comsumed by a compute job, it is not available for osd caches
[17:20] <Be-El> performance will degrade in that situation
[17:21] * kawa2014 (~kawa@2001:67c:1560:8007::aac:c1a6) Quit (Ping timeout: 480 seconds)
[17:22] * DV_ (~veillard@2001:41d0:a:f29f::1) Quit (Ping timeout: 480 seconds)
[17:22] <peem_> rlrevell: I'm using hammer as far as I know, but it still is not working well for me. Main problem is that there is so many parts on the way I'm not sure where to debug it. For example, right now radosgw doesn't want to start, while all I did is rebooted the servers.
[17:22] <doppelgrau> perpetualrabbit: If you mix compute and osd on the same host, I would recommend to put the compute part in a virtual server (and assign a relativly low CPU-priority to it), so the OSDs have a constant amount of RAM and sufficient CPU-power
[17:23] <rlrevell> peem_: what guide did you follow to deploy it?
[17:23] <perpetualrabbit> If I set a limit for rss in /etc/security/limits.conf. Say no more than 90% of the total memory of 128GiB for compute users, would that work.
[17:23] <perpetualrabbit> ?
[17:25] * Vacuum_ (~Vacuum@i59F790E8.versanet.de) Quit (Quit: leaving)
[17:25] * Vacuum_ (~Vacuum@i59F790E8.versanet.de) has joined #ceph
[17:25] <perpetualrabbit> doppelgrau, yes, that would be a good idea. But then it would make sense to make a complete openstack setup out of it, and define virtual clusters for different scientific projects as needed.
[17:25] * yguang11 (~yguang11@nat-dip30-wl-d.cfw-a-gci.corp.yahoo.com) has joined #ceph
[17:26] * Hemanth (~Hemanth@117.192.249.144) has joined #ceph
[17:26] * shnarch (~shnarch@2.52.61.121) has joined #ceph
[17:26] <peem_> rlrevell: not sure now, as I seems to have closed that tab, but mainly the one one ceph.org page.. mainly, as I used ceph-deploy to install and create it, not yum.
[17:26] <rlrevell> peem_: what distro
[17:26] * yanzheng (~zhyan@171.216.94.129) Quit (Quit: This computer has gone to sleep)
[17:27] * yanzheng (~zhyan@171.216.94.129) has joined #ceph
[17:27] <peem_> rlrevell: CentOs 7
[17:27] * moore (~moore@64.202.160.88) has joined #ceph
[17:28] * yanzheng (~zhyan@171.216.94.129) Quit ()
[17:28] <rlrevell> peem_: i'm on ubuntu and it starts for me. might be distro specific
[17:28] <peem_> rlrevell : found it : http://ceph.com/docs/master/radosgw/config/.
[17:28] * dgurtner_ (~dgurtner@178.197.236.107) Quit (Read error: Connection reset by peer)
[17:28] * shnarch (~shnarch@2.52.61.121) Quit (Read error: Connection reset by peer)
[17:29] <peem_> Anyway, I'll start over...
[17:29] * kefu (~kefu@60.247.111.66) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[17:29] <rlrevell> peem_: i don't think all that should still be needed. there's a bunch of stuff that refers to apache which is no longer used in new versions. try just going by http://ceph.com/docs/master/start/quick-ceph-deploy/#add-an-rgw-instance
[17:31] * kefu (~kefu@60.247.111.66) has joined #ceph
[17:31] * kefu (~kefu@60.247.111.66) Quit ()
[17:32] * DV (~veillard@2001:41d0:1:d478::1) has joined #ceph
[17:32] * treenerd (~treenerd@85.193.140.98) Quit (Remote host closed the connection)
[17:33] * dicko (~arsenaali@9S0AAAWHA.tor-irc.dnsbl.oftc.net) Quit ()
[17:34] * dgurtner (~dgurtner@178.197.236.107) has joined #ceph
[17:35] <peem_> rlrevell: that's how I started, but no luck - service fails to start...
[17:35] * xarses (~andreww@50.141.6.2) Quit (Ping timeout: 480 seconds)
[17:41] * tupper_ (~tcole@173.38.117.67) Quit (Read error: Connection reset by peer)
[17:43] <peem_> rlrevell: ceph-deploy works, but ceph-radosgw complains about missing keyring file....
[17:43] <perpetualrabbit> Be-El, I think I must conclude that ceph is not for us in this situation. The solution I see for mixing compute and osd on the cluster would be to make it a full openstack cluster, where the users only have access to (rather big but strictly limited) virtual machines. Then the osds would have guaranteed resources. But I have no mandate to change to cluster this way at this time although it would be cool to try. I am going to try
[17:43] <perpetualrabbit> glusterfs now, and see it that is any different. Although I fear that the same issues will be there too
[17:44] * xarses (~andreww@50.141.4.5) has joined #ceph
[17:44] <rlrevell> peem_: i did actually do the keyring related steps from the first link you posted iirc
[17:45] * joshd1 (~jdurgin@68-119-140-18.dhcp.ahvl.nc.charter.com) Quit (Quit: Leaving.)
[17:45] * alfredodeza (~alfredode@198.206.133.89) has left #ceph
[17:46] <peem_> Is there a command in ceph-deploy to nuke rgw setup ?
[17:53] <peem_> rlrevell: by not doing apache related steps, you mean ignoring socket setting in ceph conf file too ?
[17:53] <rlrevell> yeah that shouldn't be needed
[17:54] <rlrevell> oh wait, the socket setting is there
[17:55] <rlrevell> this is all i need to get it to work http://pastebin.com/veYKh0sg
[17:56] * imjustmatthew (~imjustmat@pool-74-110-227-240.rcmdva.fios.verizon.net) Quit (Ping timeout: 480 seconds)
[17:56] * xarses (~andreww@50.141.4.5) Quit (Ping timeout: 480 seconds)
[17:57] * ade (~abradshaw@tmo-101-101.customers.d1-online.com) Quit (Quit: Too sexy for his shirt)
[17:59] * jashank42 (~jashan42@117.197.168.193) has joined #ceph
[17:59] * cholcombe (~chris@c-73-180-29-35.hsd1.or.comcast.net) has joined #ceph
[17:59] * owasserm (~owasserm@216.1.187.164) Quit (Ping timeout: 480 seconds)
[18:02] <perpetualrabbit> Be-El, do you agree with my conclusion?
[18:03] * jwilkins (~jwilkins@2601:9:4580:f4c:ea2a:eaff:fe08:3f1d) Quit (Ping timeout: 480 seconds)
[18:04] * Jyron (~Qiasfah@static-ip-85-25-103-119.inaddr.ip-pool.com) has joined #ceph
[18:04] * wenjunhuang__ (~wenjunhua@61.135.172.68) has joined #ceph
[18:04] * bandrus (~brian@131.sub-70-211-67.myvzw.com) has joined #ceph
[18:06] * i_m (~ivan.miro@deibp9eh1--blueice1n2.emea.ibm.com) Quit (Ping timeout: 480 seconds)
[18:11] * diegows (~diegows@190.190.5.238) Quit (Ping timeout: 480 seconds)
[18:11] * wenjunhuang_ (~wenjunhua@61.135.172.68) Quit (Ping timeout: 480 seconds)
[18:11] * visbits (~textual@8.29.138.28) has joined #ceph
[18:12] <visbits> running into an issue provisioning new osds
[18:12] <visbits> http://pastebin.com/TdH6cSgL
[18:13] * zack_dolby (~textual@pa3b3a1.tokynt01.ap.so-net.ne.jp) has joined #ceph
[18:16] * evilrob00 (~evilrob00@cpe-72-179-3-209.austin.res.rr.com) Quit (Read error: Connection reset by peer)
[18:16] * evilrob00 (~evilrob00@cpe-72-179-3-209.austin.res.rr.com) has joined #ceph
[18:16] * bene (~ben@2601:18c:8300:2e88:56ee:75ff:fe10:7266) Quit (Quit: Konversation terminated!)
[18:17] * evilrob00 (~evilrob00@cpe-72-179-3-209.austin.res.rr.com) Quit (Read error: Connection reset by peer)
[18:18] * evilrob00 (~evilrob00@cpe-72-179-3-209.austin.res.rr.com) has joined #ceph
[18:26] <jidar> running into an odd issue with osd activate, "ceph-disk: Cannot discover filesystem type: device /dev/sdb: Line is truncated:" see full log here: http://pastebin.com/1PiHzJed
[18:26] <jidar> any ideas? I was able to do a ceph-deploy osd prepare without an issue, and disk-list comes back fine
[18:27] <jidar> one thing that I'm a little confused about is what steps you perform on what hosts, I'm doing this on openstack so I've got my mon hosts setup on my controllers, ctrl{1,2,3} and my osd hosts are ceph{1,2,3}
[18:29] * jrankin (~jrankin@nat-pool-bos-t.redhat.com) has joined #ceph
[18:33] * Jyron (~Qiasfah@5NZAADIIR.tor-irc.dnsbl.oftc.net) Quit ()
[18:33] * SaneSmith (~hifi@85.17.24.8) has joined #ceph
[18:37] * Hemanth (~Hemanth@117.192.249.144) Quit (Ping timeout: 480 seconds)
[18:37] * lcurtis (~lcurtis@47.19.105.250) has joined #ceph
[18:37] * oro (~oro@2001:620:20:16:159d:8f9e:2ff5:5914) Quit (Ping timeout: 480 seconds)
[18:38] * jordanP (~jordan@scality-jouf-2-194.fib.nerim.net) Quit (Quit: Leaving)
[18:39] * ngoswami_ (~ngoswami@121.244.87.116) has joined #ceph
[18:40] * Hemanth (~Hemanth@117.213.181.73) has joined #ceph
[18:43] * ngoswami (~ngoswami@121.244.87.116) Quit (Ping timeout: 480 seconds)
[18:51] * wschulze1 (~wschulze@cpe-69-206-240-164.nyc.res.rr.com) has joined #ceph
[18:54] <visbits> ended up being the fact i had no default root
[18:58] * jwilkins (~jwilkins@2600:1010:b062:68f1:ea2a:eaff:fe08:3f1d) has joined #ceph
[18:59] * wschulze (~wschulze@cpe-69-206-240-164.nyc.res.rr.com) Quit (Ping timeout: 480 seconds)
[19:01] <visbits> how do you move a host without editing the crush via file
[19:03] * SaneSmith (~hifi@85.17.24.8) Quit ()
[19:03] * Unforgiven (~Xerati@5NZAADINK.tor-irc.dnsbl.oftc.net) has joined #ceph
[19:03] * linuxkidd (~linuxkidd@207.236.250.131) Quit (Ping timeout: 480 seconds)
[19:03] * rotbeard (~redbeard@x5f74c14d.dyn.telefonica.de) has joined #ceph
[19:04] * Hemanth (~Hemanth@117.213.181.73) Quit (Ping timeout: 480 seconds)
[19:06] * ngoswami_ (~ngoswami@121.244.87.116) Quit (Quit: Leaving)
[19:06] * fsimonce (~simon@host253-71-dynamic.3-87-r.retail.telecomitalia.it) has joined #ceph
[19:06] * Hemanth (~Hemanth@117.192.247.117) has joined #ceph
[19:07] * kevinc (~kevinc__@client64-106.sdsc.edu) has joined #ceph
[19:07] <doppelgrau> visbits: http://ceph.com/docs/master/rados/operations/crush-map/#move-a-bucket
[19:12] <visbits> thanks that worked :)
[19:15] * linuxkidd (~linuxkidd@207.236.250.131) has joined #ceph
[19:15] * dgurtner (~dgurtner@178.197.236.107) Quit (Ping timeout: 480 seconds)
[19:18] <visbits> appears to be an issue setting the weight for disk larger than 3t?
[19:18] <visbits> weight is added as -3.052e-05
[19:18] * sleinen (~Adium@2001:620:0:2d:7ed1:c3ff:fedc:3223) Quit (Quit: Leaving.)
[19:19] * dneary (~dneary@nat-pool-bos-u.redhat.com) Quit (Ping timeout: 480 seconds)
[19:19] <doppelgrau> visbits: I???d no problems with weight > 3.0
[19:20] * adrian15b (~kvirc@btactic.ddns.jazztel.es) Quit (Ping timeout: 480 seconds)
[19:20] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[19:21] <doppelgrau> visbits: or did you set the weight for >3YB
[19:21] <visbits> 3YB?
[19:21] <visbits> using the prepare function of ceph-deploy
[19:22] <doppelgrau> Yottabyte :)
[19:23] * ircolle (~ircolle@66-194-8-225.static.twtelecom.net) has joined #ceph
[19:24] <doppelgrau> weight 1.0 is recommended for 1 TB, if you set the weight to (2^10)^4 for TB I???ll guess there could be easily an overflow
[19:26] * mykola (~Mikolaj@91.225.200.209) has joined #ceph
[19:27] * owasserm (~owasserm@206.169.83.146) has joined #ceph
[19:28] * puffy (~puffy@c-50-131-179-74.hsd1.ca.comcast.net) has joined #ceph
[19:28] <jidar> http://tracker.ceph.com/issues/7598
[19:28] <jidar> looks like I'm running into this
[19:29] <jidar> I'm going to try zapping and then osd create, because that worked last time
[19:29] <visbits> parted the disk and mklabel gpt and rm any partitions
[19:31] * wicope (~wicope@0001fd8a.user.oftc.net) Quit (Read error: Connection reset by peer)
[19:31] <rotbeard> visbits, in my dumpling and hammer testcluster I have OSDs with a weight of 4. no problems here
[19:31] * stiopa (~stiopa@cpc73828-dals21-2-0-cust630.20-2.cable.virginm.net) has joined #ceph
[19:31] <visbits> did you add them via ceph-deploy prepare?
[19:32] <rotbeard> visbits, for the new hammer cluster -> yes. for the dumpling one -> nope, manually
[19:32] <visbits> interesting
[19:33] * Unforgiven (~Xerati@5NZAADINK.tor-irc.dnsbl.oftc.net) Quit ()
[19:33] * spate (~Kayla@justus.impium.de) has joined #ceph
[19:33] <rotbeard> using ubuntu 12.04 and 14.04 here
[19:37] * thomnico (~thomnico@2a01:e35:8b41:120:a996:3c09:b684:4b7a) Quit (Ping timeout: 480 seconds)
[19:42] <mongo> jidar: Just curious, why would you deploy new systems on rhel 6?
[19:42] * ircolle (~ircolle@66-194-8-225.static.twtelecom.net) Quit (Ping timeout: 480 seconds)
[19:43] <jidar> I'm not
[19:43] <jidar> this is 7
[19:44] <mongo> what backing filesystem?
[19:44] <jidar> xfs, all defaults
[19:44] <jidar> I just ran these two commands, in this order
[19:44] <jidar> ceph-deploy --overwrite-conf osd prepare ceph{1,2,3}:sd{b,c,d,e,f,g,h,i,j}:sdk
[19:44] <jidar> ceph-deploy --overwrite-conf osd activate ceph{1,2,3}:sd{b,c,d,e,f,g,h,i,j}:sdk
[19:45] <jidar> and the activate fails,
[19:45] <jidar> however
[19:45] <mongo> you are sharing a log device, that won't work.
[19:45] <jidar> if I do `osd create` everything else the same it works fine
[19:46] <jidar> the docs don't seem to have a problem with it: https://access.redhat.com/beta/documentation/en/red-hat-ceph-storage-123-installation-guide-for-rhel-x86-64#setting-up-your-administration-server
[19:46] <mongo> you need to manually partition the journal disk and specify it for each drive.
[19:46] <jidar> /dev/sdk :
[19:46] <jidar> /dev/sdk1 ceph journal, for /dev/sdb1
[19:46] <jidar> /dev/sdk2 ceph journal, for /dev/sdc1
[19:46] <jidar> /dev/sdk3 ceph journal, for /dev/sdd1
[19:46] <jidar> it does this for you
[19:47] <jidar> both the osd prepare and the osd create do that
[19:47] <mongo> yes, ceph deploy never worked for me.
[19:47] <mongo> but passing the full dev and part did, but I am on ubuntu
[19:48] <jidar> I have a feeling the similarities of what we're doing begin and end with 'ceph' and 'computers' :P
[19:48] <mongo> heh :)
[19:48] <jidar> this is on UCS gear, single raid 0 devices (which makes me sad)
[19:49] <mongo> well I need to avoid bit rot, btrfs is not usable on redhat's vintage kernel.
[19:49] <jidar> but these stupid boxes come with raid controllers I can't get around
[19:49] <mongo> ya, it is nice to be on hba's or sata.
[19:49] * TMM (~hp@178-84-46-106.dynamic.upc.nl) has joined #ceph
[19:49] * rdas (~rdas@121.244.87.116) Quit (Quit: Leaving)
[19:49] <mongo> I had to fight to buy correct gear, because we are a dell shop so have that same issue.
[19:52] * Hemanth (~Hemanth@117.192.247.117) Quit (Ping timeout: 480 seconds)
[19:53] * oro (~oro@84-72-20-79.dclient.hispeed.ch) has joined #ceph
[19:53] <jidar> I know that struggle very well
[19:53] <jidar> hahah
[19:54] * Hemanth (~Hemanth@117.221.98.216) has joined #ceph
[19:55] <mongo> I did just order a bunch of QuantaGrid D51PH-1ULH which start shipping next month, if the ssd's supported HVMe they would be the ultimate node.
[19:56] <mongo> http://www.storagereview.com/qct_announces_three_new_storage_servers <- 12 hot swap 3.5" + 4 hot swap 2.5"
[19:57] * xarses (~andreww@156.39.10.45) has joined #ceph
[19:58] <jidar> oh get this
[19:58] <jidar> I've got SSD's in these OSD hosts I can't even use
[19:58] <jidar> because the UCS gear has specific drivers available from the vendor
[19:58] <jidar> that don't work with rhel7
[19:59] <jidar> very, very nice SSDs too, PCIe flash cards :(
[19:59] <mongo> and you are stuck with rhel?
[19:59] <jidar> I am, this is a openstack deployment
[19:59] <jidar> so the whole stack has to be from the osp installer
[20:00] * wenjunhuang__ (~wenjunhua@61.135.172.68) Quit (Remote host closed the connection)
[20:00] * wenjunhuang__ (~wenjunhua@61.135.172.68) has joined #ceph
[20:00] <mongo> oh man, being stuck on 3.10 is painful for virtualization. I feel your pain.
[20:01] <visbits> run kernel-el
[20:01] <mongo> I wonder how they do that with osp, qemu in the normal rhel 7 doesn't even have rbd support.
[20:01] <visbits> http://elrepo.org/tiki/kernel-ml
[20:01] <jidar> well, that's not really our problem. The deployment methods are just awful
[20:01] <visbits> actually the latest centos plus has rbd
[20:01] <visbits> you can thank me
[20:02] <mongo> visbits: I'll just be happy our openstack is ubuntu based, and that they backport kernels
[20:02] <visbits> https://bugs.centos.org/view.php?id=7372
[20:02] <jidar> yea, just because the kernel is 3.10 for rhel doesn't mean it doesn't include 3.10+newer patches
[20:02] <mongo> patches, not functonality though.
[20:02] <jidar> no, functionality as well
[20:02] <jidar> see visbits :)
[20:02] <gleam> mongo: give us an updat ewhen those quantagrids come in
[20:03] <gleam> they look fantastic
[20:03] <gleam> you've used those asus hadoop nodes too right?
[20:03] * spate (~Kayla@8Q4AABEGC.tor-irc.dnsbl.oftc.net) Quit ()
[20:03] <visbits> you guys should check out EMC ScaleIO
[20:03] * Neon (~Enikma@89.105.194.89) has joined #ceph
[20:03] <visbits> if you need block performance it makes ceph look like garbage
[20:03] <mongo> gleam: yes, the asus ones are painful to have remote hands work on.
[20:03] <jidar> for about a million dollars ;)
[20:03] <visbits> its free
[20:04] <mongo> gleam: I also have supermicro 4xsled systems in our pure ssd based system.
[20:04] * dneary (~dneary@nat-pool-bos-u.redhat.com) has joined #ceph
[20:04] <mongo> gleam: quanta is quite solid, they are the ones that design and build dell/hp/cisco servers.
[20:04] <gleam> yep
[20:05] * ircolle (~ircolle@206.169.83.146) has joined #ceph
[20:07] <mongo> visbits: I would be suspect of any distributed system that is lots faster than ceph, not that ceph doesn't have places to improve but it probably means they are being risky with data.
[20:07] <visbits> ceph is just fat
[20:09] <mongo> most of the latency in our system is the replica writes, the only way EMC can make that faster is by not ensuring multiple replicas before acking a write.
[20:10] <mongo> Oh, I see it smokes ceph if you are writing data to /dev/null
[20:10] * Hemanth (~Hemanth@117.221.98.216) Quit (Ping timeout: 480 seconds)
[20:11] * Hemanth (~Hemanth@117.192.234.109) has joined #ceph
[20:11] <koollman> but you can replace it with a cheaper alternative, http://devnull-as-a-service.com/ :)
[20:11] * yguang11 (~yguang11@nat-dip30-wl-d.cfw-a-gci.corp.yahoo.com) Quit (Remote host closed the connection)
[20:11] * yguang11 (~yguang11@2001:4998:effd:600:e891:ea14:b21b:18c7) has joined #ceph
[20:14] * TheSov2 (~TheSov@204.13.200.248) has joined #ceph
[20:14] * midnightrunner (~midnightr@216.113.160.65) has joined #ceph
[20:14] <TheSov2> if I wanted to purchase a support contract for ceph. who would offer that?
[20:15] <TheSov2> redhat?
[20:17] <jidar> they offer support ceph
[20:17] <jidar> but only on their stack (iirc)
[20:18] <gregsfortytwo> Canonical just (re)launched support as well, and Suse has offered it in their storage system for several years
[20:18] <jidar> what tells a mon-host what osd's the osd-host has?
[20:18] <gregsfortytwo> I'm a little biased towards the Red Hat offering, though ;)
[20:18] <TheSov2> good, dell's support is only if you use their hardware vendor
[20:18] * Kupo1 (~tyler.wil@23.111.254.159) has joined #ceph
[20:18] <TheSov2> and i hate being hardware locked
[20:19] <jidar> not much different from being software locked hahaha
[20:19] <TheSov2> the software is opensource and free
[20:19] <TheSov2> vendors cost money
[20:19] <jidar> so does support
[20:19] <TheSov2> yes
[20:20] <TheSov2> but its nice to be able to shop around
[20:20] * ChrisNBlum (~ChrisNBlu@dhcp-ip-230.dorf.rwth-aachen.de) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[20:20] <jidar> you're trading what problem for another is all I'm saying
[20:20] <jidar> s/what/one
[20:20] <TheSov2> im mostly going to support this on my own, but my company wants fallback support
[20:20] <TheSov2> best bang for the buck is super micro
[20:20] <TheSov2> i dont intend to pass that up
[20:20] <jidar> so I've got a bunch of these " /dev/sdb1 ceph data, prepared, cluster ceph, journal /dev/sdk1
[20:21] <jidar> on my osd hosts, but my `ceph osd tree` is empty on my mon hosts
[20:21] <jidar> the ceph.conf on the osd host is pointing to the mon hosts
[20:21] * dneary (~dneary@nat-pool-bos-u.redhat.com) Quit (Ping timeout: 480 seconds)
[20:21] <jidar> is it normal to not be able to run `ceph osd tree` on your osd hosts?
[20:23] * derjohn_mob (~aj@fw.gkh-setu.de) Quit (Ping timeout: 480 seconds)
[20:24] * xarses (~andreww@156.39.10.45) Quit (Ping timeout: 480 seconds)
[20:26] * kklimonda (~kklimonda@2001:41d0:a:f2b5::1) Quit (Ping timeout: 480 seconds)
[20:26] * kklimonda (~kklimonda@2001:41d0:a:f2b5::1) has joined #ceph
[20:33] * Neon (~Enikma@5NZAADITD.tor-irc.dnsbl.oftc.net) Quit ()
[20:33] * Aethis (~smf68@torland1-this.is.a.tor.exit.server.torland.is) has joined #ceph
[20:33] * evilrob00 (~evilrob00@cpe-72-179-3-209.austin.res.rr.com) Quit (Read error: Connection reset by peer)
[20:33] * evilrob00 (~evilrob00@cpe-72-179-3-209.austin.res.rr.com) has joined #ceph
[20:35] * rotbeard (~redbeard@x5f74c14d.dyn.telefonica.de) Quit (Quit: Leaving)
[20:35] * sleinen (~Adium@84-72-160-233.dclient.hispeed.ch) has joined #ceph
[20:37] * midnight_ (~midnightr@216.113.160.71) has joined #ceph
[20:37] * ira (~ira@c-71-233-225-22.hsd1.ma.comcast.net) Quit (Quit: Leaving)
[20:38] * sleinen1 (~Adium@2001:620:0:82::101) has joined #ceph
[20:38] * ira (~ira@c-71-233-225-22.hsd1.ma.comcast.net) has joined #ceph
[20:39] * midnightrunner (~midnightr@216.113.160.65) Quit (Ping timeout: 480 seconds)
[20:43] * elder (~elder@c-24-245-18-91.hsd1.mn.comcast.net) Quit (Ping timeout: 480 seconds)
[20:43] * sleinen (~Adium@84-72-160-233.dclient.hispeed.ch) Quit (Ping timeout: 480 seconds)
[20:44] * scuttle|afk is now known as scuttlemonkey
[20:47] * adrian15b (~kvirc@26.Red-83-49-245.dynamicIP.rima-tde.net) has joined #ceph
[20:52] <mongo> TheSov2, supermicro resellers like silicon mechanics may offer ceph support.
[20:52] * elder (~elder@c-24-245-18-91.hsd1.mn.comcast.net) has joined #ceph
[20:52] * ChanServ sets mode +o elder
[20:53] <mongo> jidar: I prefer to restrict the number of hosts which have the admin key, `ceph osd tree` will try to use that key by default.
[20:54] <jidar> mongo: hmm, ok - anything to say about why my OSD's arn't seen by my mon hosts?
[20:54] <mongo> gleam: the other issue with the asus hosts, not enough ports to boot from a satadom, the quanta systems support that which will simplify life.
[20:55] <mongo> jidar: you deployed /etc/ceph/ceph.conf to all hosts right? if it is there typically the OSDs aren't started or are having issues starting but that should be logged to a file in /var/log/ceph
[20:56] * Be-El (~quassel@fb08-bcf-pc01.computational.bio.uni-giessen.de) Quit (Remote host closed the connection)
[20:57] <jidar> hmm, let me get this puppet interview out of the way and I'll take a look there
[20:58] * jwilkins (~jwilkins@2600:1010:b062:68f1:ea2a:eaff:fe08:3f1d) Quit (Ping timeout: 480 seconds)
[21:01] * Hemanth (~Hemanth@117.192.234.109) Quit (Quit: Leaving)
[21:02] * rwheeler (~rwheeler@nat-pool-bos-t.redhat.com) Quit (Quit: Leaving)
[21:03] * Aethis (~smf68@9S0AAAWSM.tor-irc.dnsbl.oftc.net) Quit ()
[21:03] * linjan (~linjan@46.210.246.8) has joined #ceph
[21:05] * midnightrunner (~midnightr@216.113.160.65) has joined #ceph
[21:06] * midnightrunner (~midnightr@216.113.160.65) Quit (Remote host closed the connection)
[21:06] * linjan (~linjan@46.210.246.8) Quit (Read error: Connection reset by peer)
[21:07] * midnightrunner (~midnightr@216.113.160.65) has joined #ceph
[21:10] * alfredodeza (~alfredode@198.206.133.89) has joined #ceph
[21:12] * nsoffer (~nsoffer@bzq-79-180-80-86.red.bezeqint.net) has joined #ceph
[21:12] * davidz (~davidz@2605:e000:1313:8003:79d9:2a2:f03f:51ae) has joined #ceph
[21:13] * midnight_ (~midnightr@216.113.160.71) Quit (Ping timeout: 480 seconds)
[21:13] * midnight_ (~midnightr@216.113.160.71) has joined #ceph
[21:14] * davidz1 (~davidz@cpe-23-242-27-128.socal.res.rr.com) Quit (Ping timeout: 480 seconds)
[21:17] * smartattack (~petem@192.81.213.49) has joined #ceph
[21:17] * midnightrunner (~midnightr@216.113.160.65) Quit (Ping timeout: 480 seconds)
[21:20] * dephcon (~oftc-webi@c73-110.rim.net) Quit (Remote host closed the connection)
[21:22] * kevinc (~kevinc__@client64-106.sdsc.edu) Quit (Quit: Leaving)
[21:23] * bene (~ben@c-24-60-237-191.hsd1.nh.comcast.net) has joined #ceph
[21:26] * ramonskie (ab15507e@107.161.19.53) has joined #ceph
[21:28] * rotbeard (~redbeard@2a02:908:df10:d300:76f0:6dff:fe3b:994d) has joined #ceph
[21:32] * daviddcc (~dcasier@84.197.151.77.rev.sfr.net) has joined #ceph
[21:33] * w2k (~Zeis@199.58.87.2) has joined #ceph
[21:37] * wicope (~wicope@0001fd8a.user.oftc.net) has joined #ceph
[21:38] <ramonskie> i'm trying to add a monitor with ceph-deploy mon add host
[21:38] <ramonskie> but i get the following error cephx: verify_reply couldn't decrypt with error: error decoding block for decryption
[21:40] * arbrandes (~arbrandes@177.138.114.132) Quit (Remote host closed the connection)
[21:43] * sleinen1 (~Adium@2001:620:0:82::101) Quit (Ping timeout: 480 seconds)
[21:45] * garphy`aw is now known as garphy
[21:49] * linuxkidd (~linuxkidd@207.236.250.131) Quit (Quit: Leaving)
[21:51] * dupont-y (~dupont-y@familledupont.org) has joined #ceph
[21:53] <visbits> lol 177tb pool and i just changed the pgp numbers.. 3 days later
[21:59] * linjan (~linjan@80.179.241.26) has joined #ceph
[22:01] * TheSov2 (~TheSov@204.13.200.248) Quit (Read error: Connection reset by peer)
[22:03] * w2k (~Zeis@7R2AABLBG.tor-irc.dnsbl.oftc.net) Quit ()
[22:03] * n0x1d (~Rosenblut@8Q4AABEKX.tor-irc.dnsbl.oftc.net) has joined #ceph
[22:04] * dupont-y (~dupont-y@familledupont.org) Quit (Read error: No route to host)
[22:05] * dupont-y (~dupont-y@familledupont.org) has joined #ceph
[22:07] * adrian15b (~kvirc@26.Red-83-49-245.dynamicIP.rima-tde.net) Quit (Ping timeout: 480 seconds)
[22:08] * sjm (~sjm@pool-173-70-76-86.nwrknj.fios.verizon.net) has left #ceph
[22:08] * sjm (~sjm@pool-173-70-76-86.nwrknj.fios.verizon.net) has joined #ceph
[22:09] * jwilkins (~jwilkins@2601:9:4580:f4c:ea2a:eaff:fe08:3f1d) has joined #ceph
[22:14] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit (Ping timeout: 480 seconds)
[22:16] * lovejoy (~lovejoy@cpc69388-oxfd28-2-0-cust415.4-3.cable.virginm.net) has joined #ceph
[22:17] * lovejoy (~lovejoy@cpc69388-oxfd28-2-0-cust415.4-3.cable.virginm.net) Quit ()
[22:20] <ramonskie> is there no one who can help me out with my add mon issue?
[22:21] * derjohn_mob (~aj@p578b6aa1.dip0.t-ipconnect.de) has joined #ceph
[22:23] * garphy is now known as garphy`aw
[22:31] * alfredodeza (~alfredode@198.206.133.89) has left #ceph
[22:33] * n0x1d (~Rosenblut@8Q4AABEKX.tor-irc.dnsbl.oftc.net) Quit ()
[22:33] * Uniju1 (~Xa@176.10.99.200) has joined #ceph
[22:33] * dyasny (~dyasny@173.231.115.59) Quit (Ping timeout: 480 seconds)
[22:33] * CheKoLyN (~saguilar@bender.parc.xerox.com) has joined #ceph
[22:36] * Sysadmin88 (~IceChat77@054527d3.skybroadband.com) has joined #ceph
[22:36] * dupont-y (~dupont-y@familledupont.org) Quit (Quit: Ex-Chat)
[22:37] * yguang11 (~yguang11@2001:4998:effd:600:e891:ea14:b21b:18c7) Quit (Remote host closed the connection)
[22:37] * yguang11 (~yguang11@nat-dip30-wl-d.cfw-a-gci.corp.yahoo.com) has joined #ceph
[22:37] * zaitcev (~zaitcev@2001:558:6001:10:61d7:f51f:def8:4b0f) has joined #ceph
[22:38] * oro (~oro@84-72-20-79.dclient.hispeed.ch) Quit (Ping timeout: 480 seconds)
[22:39] * badone (~brad@CPE-121-215-241-179.static.qld.bigpond.net.au) has joined #ceph
[22:41] * championofcyrodi (~championo@50-205-35-98-static.hfc.comcastbusiness.net) has joined #ceph
[22:45] * yguang11 (~yguang11@nat-dip30-wl-d.cfw-a-gci.corp.yahoo.com) Quit (Ping timeout: 480 seconds)
[22:46] * championofcyrodi (~championo@50-205-35-98-static.hfc.comcastbusiness.net) has left #ceph
[22:47] * mykola (~Mikolaj@91.225.200.209) Quit (Quit: away)
[22:56] * sjm (~sjm@pool-173-70-76-86.nwrknj.fios.verizon.net) has left #ceph
[22:59] <jidar> hmmm, all I have in my /var/log/ceph/*.log is a OSD log, from when I created the OSD's a few hours ago
[22:59] <jidar> 2015-06-09 12:53:21.870853 7f59e4bef7c0 1 journal _open /dev/sdk7 fd 4: 5368709120 bytes, block size 4096 bytes, directio = 0, aio = 0
[22:59] <jidar> stuff like this
[22:59] * ghartz_ (~ghartz@AStrasbourg-651-1-162-51.w90-6.abo.wanadoo.fr) Quit (Read error: Connection reset by peer)
[23:01] * rlrevell (~leer@vbo1.inmotionhosting.com) Quit (Ping timeout: 480 seconds)
[23:03] * Uniju1 (~Xa@5NZAADI4D.tor-irc.dnsbl.oftc.net) Quit ()
[23:03] * AotC (~JamesHarr@edwardsnowden2.torservers.net) has joined #ceph
[23:03] * JFQ (~ghartz@AStrasbourg-651-1-162-51.w90-6.abo.wanadoo.fr) has joined #ceph
[23:04] * ramonskie (ab15507e@107.161.19.53) Quit (Quit: http://www.kiwiirc.com/ - A hand crafted IRC client)
[23:05] * ramonskie (d97ae1ba@107.161.19.53) has joined #ceph
[23:09] * badone_ (~brad@CPE-121-215-241-179.static.qld.bigpond.net.au) has joined #ceph
[23:10] * jwilkins (~jwilkins@2601:9:4580:f4c:ea2a:eaff:fe08:3f1d) Quit (Remote host closed the connection)
[23:12] * brad_mssw (~brad@66.129.88.50) Quit (Quit: Leaving)
[23:13] * LeaChim (~LeaChim@host86-175-32-176.range86-175.btcentralplus.com) has joined #ceph
[23:14] <jidar> anybody know how the mon_hosts and mon_initial_members is used? I ask because I'm trying to determine if my OSD hosts are using the right IP, and the way it looks the IP is listed as the mon_hosts but not the mon_initial_members
[23:15] * badone (~brad@CPE-121-215-241-179.static.qld.bigpond.net.au) Quit (Ping timeout: 480 seconds)
[23:16] * georgem1 (~Adium@fwnat.oicr.on.ca) Quit (Quit: Leaving.)
[23:20] * jwilkins (~jwilkins@2601:9:4580:f4c:ea2a:eaff:fe08:3f1d) has joined #ceph
[23:21] * marrusl (~mark@nat-pool-rdu-u.redhat.com) Quit (Remote host closed the connection)
[23:21] <ganesh> Hi
[23:22] * daviddcc (~dcasier@84.197.151.77.rev.sfr.net) Quit (Ping timeout: 480 seconds)
[23:22] <ganesh> I want to know how to pass rbd through to virt-install
[23:22] <ganesh> *rbd as argument to virt-install
[23:26] <TheSov> hello, i deployed a test environment with seperate monitors from osd systems
[23:26] <TheSov> when i tried to gatherkeys it errors
[23:26] <TheSov> so i had to purge and do a ceph-deploy mon1 mon2 mon3
[23:26] <TheSov> and it works
[23:26] <TheSov> but how do i add my osd servers now?
[23:28] * jrankin (~jrankin@nat-pool-bos-t.redhat.com) Quit (Quit: Leaving)
[23:28] * daviddcc (~dcasier@84.197.151.77.rev.sfr.net) has joined #ceph
[23:29] <jidar> if they have a ceph.conf file that has the monitor nodes in there you can prep/and or activate and osd install the disks
[23:30] <jidar> this is what I was doing: ceph-deploy --overwrite-conf osd prepare ceph{1,2,3}:sd{b,c,d,e,f,g,h,i,j}:sdk
[23:30] <jidar> ceph-deploy --overwrite-conf osd activate ceph{1,2,3}:sd{b,c,d,e,f,g,h,i,j}:sdk
[23:30] <TheSov> oh so i can do the ceph osd prepare on nodes that arent part of the "new"
[23:30] <jidar> you can verify with the ceph-disk list command
[23:31] <jidar> not sure what 'new' is here
[23:31] <jidar> but you prepare the OSD hosts, with their to-be-used disks
[23:31] <TheSov> when you create a new cluster during initial deploy
[23:31] <TheSov> after you do ceph-deploy install $all
[23:31] <TheSov> you do a "ceph-deploy new mon1 mon2 mon3"
[23:32] <TheSov> before I used the same systems as monitors and osd
[23:32] <TheSov> but im trying to segregate that
[23:32] <TheSov> so i have to copy the keyring manually to the osd servers?
[23:32] <TheSov> ceph.conf and keyring?
[23:32] <jidar> I believe ceph-deploy with --overwrite-config will do the ceph.conf
[23:33] <jidar> the keyring I'm not so sure about, maybe a scp for him
[23:33] * AotC (~JamesHarr@9S0AAAW2C.tor-irc.dnsbl.oftc.net) Quit ()
[23:33] * Rosenbluth (~isaxi@tor-exit0.intersafeit.com) has joined #ceph
[23:33] <TheSov> i see
[23:33] <jidar> I'm having trouble with that bit
[23:38] <ramonskie> i'm trying to add a monitor with ceph-deploy mon add host
[23:38] * linjan (~linjan@80.179.241.26) Quit (Ping timeout: 480 seconds)
[23:38] <ramonskie> but i get the following error cephx: verify_reply couldn't decrypt with error: error decoding block for decryption
[23:38] <ramonskie> any ideas
[23:42] <ganesh> Any idea on passing ceph rbd image to virt-install
[23:48] * davidz1 (~davidz@cpe-23-242-27-128.socal.res.rr.com) has joined #ceph
[23:48] * pikos (~shamangeo@c-24-60-202-92.hsd1.ma.comcast.net) has joined #ceph
[23:48] * jashank42 (~jashan42@117.197.168.193) Quit (Remote host closed the connection)
[23:49] * georgem (~Adium@fwnat.oicr.on.ca) has joined #ceph
[23:50] * jashank42 (~jashan42@117.197.168.193) has joined #ceph
[23:50] * ramonskie (d97ae1ba@107.161.19.53) Quit (Quit: http://www.kiwiirc.com/ - A hand crafted IRC client)
[23:52] * davidz (~davidz@2605:e000:1313:8003:79d9:2a2:f03f:51ae) Quit (Ping timeout: 480 seconds)
[23:52] <lurbs> ganesh: Don't believe that it's currently possible. In the general virt-install case you'd create a storage pool, then reference it as '--disk vol=$POOL/$VOLUME' on the virt-install commandline but it doesn't seem to work for Ceph pools.
[23:52] <lurbs> ganesh: See: https://bugzilla.redhat.com/show_bug.cgi?id=1074169
[23:52] <lurbs> That's RedHat specific, but similar applies for other distributions.
[23:54] <TheSov> does traffic flow through the monitors, or do the monitors just establish quorum?
[23:54] <lurbs> TheSov: Data traffic goes direct to the OSDs.
[23:54] <TheSov> hmm, then why do the monitors have to be a bit hefty?
[23:55] <lurbs> Clients need to talk to the monitors to authenticate and get the current cluster map, but they're not in the data path.
[23:56] <TheSov> http://pastebin.com/R2aWapUs so this is my current setup. is it safe to say that once i do the gather keys on the deploy machine i can use it to continually add new OSD and monitors?
[23:57] * daviddcc (~dcasier@84.197.151.77.rev.sfr.net) Quit (Ping timeout: 480 seconds)
[23:59] * pikos (~shamangeo@c-24-60-202-92.hsd1.ma.comcast.net) Quit (Remote host closed the connection)
[23:59] <lurbs> TheSov: Yes, although unless it's a huge cluster 3 monitors should be sufficient.

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.