#ceph IRC Log

Index

IRC Log for 2015-06-23

Timestamps are in GMT/BST.

[0:00] * MentalRay (~MRay@MTRLPQ42-1176054809.sdsl.bell.ca) has joined #ceph
[0:00] * visbits (~textual@8.29.138.28) Quit (Quit: Textual IRC Client: www.textualapp.com)
[0:03] * georgem (~Adium@23.91.150.96) has joined #ceph
[0:03] * georgem (~Adium@23.91.150.96) Quit ()
[0:04] * xarses (~xarses@12.164.168.117) has joined #ceph
[0:04] * zaitcev (~zaitcev@2001:558:6001:10:61d7:f51f:def8:4b0f) has joined #ceph
[0:07] * vbellur (~vijay@107-1-123-195-ip-static.hfc.comcastbusiness.net) has joined #ceph
[0:08] * squizzi (~squizzi@nat-pool-rdu-t.redhat.com) Quit (Ping timeout: 480 seconds)
[0:09] * kklimonda (~kklimonda@2001:41d0:a:f2b5::1) Quit (Quit: WeeChat 0.4.2)
[0:12] * reed (~reed@75-101-54-131.dsl.static.fusionbroadband.com) Quit (Ping timeout: 480 seconds)
[0:16] * wicope (~wicope@0001fd8a.user.oftc.net) Quit (Remote host closed the connection)
[0:17] * TheSov (~TheSov@cip-248.trustwave.com) Quit (Read error: Connection reset by peer)
[0:21] * reed (~reed@ip-66-33-206-9.dreamhost.com) has joined #ceph
[0:22] * primechuck (~primechuc@host-95-2-129.infobunker.com) Quit (Remote host closed the connection)
[0:24] * dneary (~dneary@nat-pool-bos-u.redhat.com) Quit (Ping timeout: 480 seconds)
[0:25] * yanzheng (~zhyan@182.139.20.134) has joined #ceph
[0:42] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[0:44] * dyasny (~dyasny@104.158.35.152) Quit (Quit: Ex-Chat)
[0:48] * MrHeavy (~MrHeavy@pool-108-54-190-117.nycmny.fios.verizon.net) has joined #ceph
[0:54] * ska (~skatinolo@cpe-173-174-111-177.austin.res.rr.com) has joined #ceph
[1:00] * lifeboy (~roland@196.45.29.61) Quit (Ping timeout: 480 seconds)
[1:01] * wer (~wer@206-248-239-142.unassigned.ntelos.net) Quit (Ping timeout: 480 seconds)
[1:02] * wer (~wer@206-248-239-142.unassigned.ntelos.net) has joined #ceph
[1:04] * joshd (~jdurgin@206.169.83.146) Quit (Quit: Leaving.)
[1:06] <ska> Whats the best way to remove all ceph configuration from a node ?
[1:07] <Nats_> the only actualy configuration data is in /etc/ceph
[1:07] <ska> I have a lot of data in /var/lib/ceph too...
[1:08] <ska> I want to get it back to some sort of non-ceph state if possible so that I can re-build it.
[1:08] <Nats_> ah
[1:09] <Nats_> what i do (on ubuntu) is "stop ceph-all"
[1:10] <Nats_> then rm /etc/ceph/* ; rm -rf /var/lib/ceph ; mkdir /var/lib/ceph ; mkdir /var/lib/ceph/bootstrap-{osd,mon,mds}
[1:12] <Nats_> for my test cluster i have a bash script that ssh to all my nodes and does that
[1:12] * alram (~alram@206.169.83.146) Quit (Quit: leaving)
[1:23] * oro (~oro@212-40-76-145.pool.digikabel.hu) Quit (Quit: Leaving)
[1:26] <ska> I can create my filesystems on each xfs mount, but I can't start ceph-mon..
[1:27] <Nats_> i stop literally everything, purge the ceph-deploy data and start over
[1:27] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit (Ping timeout: 480 seconds)
[1:27] <Nats_> using ceph-deploy to create monitors osds etc
[1:27] <ska> failed: 'timeout 30 /usr/bin/ceph -c /etc/ceph/ceph.conf --name=osd.1 --keyring=/etc/ceph/keyring osd crush create-or-move -- 1 0.02 host=ceph-01 root=default'
[1:28] <ska> I'm doing it all manually for now.
[1:29] * OutOfNoWhere (~rpb@199.68.195.101) has joined #ceph
[1:30] <ska> I was able to get it working before but only manually.. wasn't able to automate.
[1:32] * murmur_ (~murmur@zeeb.org) has joined #ceph
[1:32] * murmur (~murmur@zeeb.org) Quit (Read error: Connection reset by peer)
[1:32] <ska> Runnig ceph-mon -d manually shows me: IO error: /var/lib/ceph/mon/ceph-admin/store.db/LOCK: No such file or directory
[1:33] * lifeboy (~roland@196.32.233.103) has joined #ceph
[1:33] <ska> http://git.io/vLF0r is the general setup
[1:34] <ska> ceph-create-keys seems to be stuck too
[1:34] * asalor (~asalor@0001ef37.user.oftc.net) Quit (Ping timeout: 480 seconds)
[1:35] * asalor (~asalor@2a00:1028:96c1:4f6a:204:e2ff:fea1:64e6) has joined #ceph
[1:35] <ska> Ok,wierd.. All of the sudden, Mon is running..
[1:35] <ska> Maybe it takes a while to sync?
[1:37] <ska> Nats, I'm using Ansible for my automations, but should be similar..
[1:38] * MentalRay (~MRay@MTRLPQ42-1176054809.sdsl.bell.ca) Quit (Ping timeout: 480 seconds)
[1:39] <Nats_> ska, i found in my 'auto build' script that i had to add a delay
[1:39] <ska> At what points?
[1:39] <Nats_> in ceph-deploy terms, i do "ceph-deploy mon create $MONITORS"
[1:40] <Nats_> then sleep for 5 seconds per monitor before "ceph-deploy gatherkeys"
[1:40] <ska> Hmmm.. Thats a lot of steps..
[1:40] <Nats_> so during monitor creation. after that the cluster is 'live' and everything else requires no delay
[1:40] <ska> I mean, hidden steps in terms of my manyual deployment.
[1:41] <Nats_> http://pastebin.com/jtXbtQqb
[1:41] * lifeboy (~roland@196.32.233.103) Quit (Ping timeout: 480 seconds)
[1:42] <Nats_> test env so i didnt have to make it robust, just tweaked the sleep until it worked
[1:42] <Nats_> presumedly could just loop until i successfully received keys
[1:43] <ska> My stuff goes southe when I do: ceph-mon --mkfs -i {{ osd_id }} --monmap /etc/ceph/monmap --keyring /etc/ceph/keyring
[1:43] <ska> Maybe its too fast.
[1:44] <ska> Ok,, looks very close to the place I'm having problems...
[1:44] <ska> I think the time I got it to work, I did it manually, one host at a time.
[1:44] * jdillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) Quit (Quit: jdillaman)
[1:46] <Nats_> sounds like its probably same problem
[1:49] * oms101 (~oms101@p20030057EA07C400C6D987FFFE4339A1.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[1:57] * oms101 (~oms101@p20030057EA06BB00C6D987FFFE4339A1.dip0.t-ipconnect.de) has joined #ceph
[2:00] * cholcombe (~chris@c-73-180-29-35.hsd1.or.comcast.net) Quit (Ping timeout: 480 seconds)
[2:00] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[2:05] * moore (~moore@64.202.160.233) Quit (Remote host closed the connection)
[2:05] * DV (~veillard@2001:41d0:a:f29f::1) has joined #ceph
[2:06] * moore (~moore@64.202.160.233) has joined #ceph
[2:08] * yguang11_ (~yguang11@2001:4998:effd:600:11f6:3c86:6c6d:dd5) Quit (Remote host closed the connection)
[2:08] * xarses (~xarses@12.164.168.117) Quit (Ping timeout: 480 seconds)
[2:09] * yguang11_ (~yguang11@2001:4998:effd:600:7ddb:1071:473b:4654) has joined #ceph
[2:10] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[2:14] * moore (~moore@64.202.160.233) Quit (Read error: Connection reset by peer)
[2:15] * nhm (~nhm@184-97-242-33.mpls.qwest.net) has joined #ceph
[2:15] * ChanServ sets mode +o nhm
[2:16] * reed (~reed@ip-66-33-206-9.dreamhost.com) Quit (Ping timeout: 480 seconds)
[2:18] * arbrandes (~arbrandes@189.78.72.151) Quit (Quit: Leaving)
[2:20] <TheSov2> so can i run hammer monitors and firefly osd's without issue?
[2:23] <Nats_> i think the recommendation is to update everything but in the short term you can
[2:23] * Debesis (0x@143.252.117.89.static.mezon.lt) Quit (Ping timeout: 480 seconds)
[2:24] <via> i've just replaced a bunch of OSDs, but they don't seem to want to join the cluster. the log just sits at https://pastee.org/j4yz3
[2:24] <via> they're still reported as out and down
[2:25] * xarses (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) has joined #ceph
[2:25] <via> stracing the osd proces shows it doing nothing, and there's no disk io
[2:27] * tupper (~tcole@173.38.117.80) Quit (Ping timeout: 480 seconds)
[2:27] * yguang11_ (~yguang11@2001:4998:effd:600:7ddb:1071:473b:4654) Quit (Remote host closed the connection)
[2:32] <TheSov2> wait
[2:32] <TheSov2> what do you mean by replaced?
[2:32] <TheSov2> as in you removed the OSD entirely and then added new ones?
[2:32] <via> the drives had failed
[2:32] <TheSov2> or that u changed the disk to an OSD without first removing it
[2:32] <via> i put new ones in, and ran ceph-osd --mkfs --mkjournal on them
[2:33] <via> which is how i've historically done it
[2:33] <TheSov2> you cannot just change a drive. you need to delete the OSD first
[2:33] <TheSov2> apparently that is no longer the case
[2:33] <via> can i do that and retain the same numbering?
[2:33] <TheSov2> if you google drive failure steps in hammer, it says to remove the OSD from auth, system and crusmap
[2:33] <TheSov2> after doing so you can replace the disk
[2:33] <TheSov2> then you redeploy the disk as a new osd.
[2:34] <via> man thats a pain in the ass compared to how i used to do it
[2:34] <TheSov2> i agree
[2:34] <TheSov2> but in the long run its best
[2:34] <TheSov2> think of it this way
[2:34] <TheSov2> its late on friday and a disk goes down, do you want to goto the DC to change it or just pull it out of the cluster and replace it at your leasure?
[2:35] <via> well, i mean, these drives have been failed for months
[2:35] <via> i've been terribly lazy
[2:35] <TheSov2> lol
[2:35] <TheSov2> thats the spirit
[2:35] <TheSov2> this way helps u plan for that
[2:35] <via> apparently osd rm'ing it and then re-creating it will be enough
[2:35] * via just did that
[2:36] * RomeroJnr (~h0m3r@hosd.leaseweb.net) Quit ()
[2:36] <via> if i do one at a time it'll preserve the same numbering
[2:36] * shyu (~Shanzhi@119.254.120.66) has joined #ceph
[2:37] <via> thanks
[2:37] <TheSov2> ceph osd out osd.99, ceph osd crush remove osd.99, ceph auth del osd.99
[2:37] <TheSov2> there you go
[2:37] <TheSov2> then your cluster should back fill the missing data
[2:37] <via> ceph osd rm 2; ceph osd create; and then a service restart was sufficient
[2:37] <TheSov2> eh?
[2:38] <via> i didn't have to remove from the crush map or auth
[2:38] <TheSov2> but then its still in your crushmap and auth... i think it will cause you issues later
[2:38] <via> well, i reuse the auth
[2:38] <via> key
[2:38] <TheSov2> ok
[2:38] <TheSov2> hey man if you know what your doing, you good :)
[2:38] <via> this is just a personal cluster, not a huge deal if it fucks up
[2:38] <TheSov2> i just started with ceph and im learning the inns and outs
[2:38] <via> i've been using it for 3.5 years
[2:39] <via> and the processes have changed a bunch over time
[2:39] <via> ceph-deploy didn't exist back then
[2:39] <TheSov2> yeah i got a problem now
[2:39] <TheSov2> i got 6 raspberry pi's i want to use as OSD servers
[2:39] <TheSov2> and 1 intel atom to use as a monitor
[2:40] <TheSov2> the repo's for ceph arm are strangely vacant....
[2:40] <TheSov2> the latest version in "testing" repo for raspberry pi is firefly
[2:40] <TheSov2> so my monitor is hammer
[2:40] * lucas (~Thunderbi@218.76.52.64) has joined #ceph
[2:40] <TheSov2> and my nodes will be firefly and that puts me in a state of unease
[2:40] <via> make that 2.5 years, not 3.5
[2:40] <via> i've found cpu to be somewhat imporant
[2:40] <via> my amd apu's are underpowred for the jobs
[2:40] <TheSov2> i need a POC for work before we build a 2PB cluster
[2:41] <via> r-pi seems woefully underpowered for this
[2:41] <TheSov2> its just to prove it will work
[2:41] <via> fair enough
[2:41] <TheSov2> im actually very surprised no one has built a purpose driven arm computer for use as an OSD
[2:42] <TheSov2> if i was an ECE i would be on this like glue
[2:42] * nsoffer (~nsoffer@bzq-79-180-80-9.red.bezeqint.net) Quit (Ping timeout: 480 seconds)
[2:42] <via> well, you can't go to newegg and buy arm motherboards last i checked
[2:42] <TheSov2> look up bananapi
[2:42] <via> ece folks like avr/pic/8051's and shit :p
[2:42] <via> not arm
[2:43] <TheSov2> ok some CE then
[2:43] <TheSov2> whatever the point is there is a marget for a drive bay that has ssd and rust both in 1 package with a gig port on the back and POE
[2:43] <TheSov2> market*
[2:43] <via> yes
[2:43] <TheSov2> stackable :)
[2:44] <TheSov2> make it ez to firmware flash for updating and you could be rich!
[2:44] <via> i'm not heavily investing in arm until we have vaguely more open video
[2:44] <TheSov2> sellem at like 100 bucks a pop
[2:44] <via> i hate the bullshit binary blobs you have to run to get even basic stuff
[2:45] <TheSov2> problem is even intel does that, no one has any clue how to turn a intel processor on
[2:45] * fam_away is now known as fam
[2:45] <TheSov2> intel gives you blobs to do it from the chipset
[2:45] <TheSov2> so to even kick on the main processor from intel you need to license their IP
[2:46] <via> i'm a little behind the times, but that must be a recent thing
[2:46] <via> unless you're just talking about microcode updates
[2:46] <TheSov2> http://www.pcworld.com/article/2883903/how-intel-and-pc-makers-prevent-you-from-modifying-your-pcs-firmware.html
[2:46] <TheSov2> it is recent
[2:47] <via> yeah, i don't have anything efi
[2:47] <TheSov2> but thats the way its going
[2:47] <via> you can turn off verified boot
[2:47] <TheSov2> recently i saw an IBM with an amazing efi, i was very impressed
[2:48] <TheSov2> it would take your hardware, and turn it into a virtual profil
[2:48] <TheSov2> broadcom nic's became intel, no one video became ati etc, etc
[2:48] <TheSov2> when you installed the OS it thought it had genuine intel nics in it
[2:49] <TheSov2> and it was shoting me an ATI video card, in lspci
[2:49] <TheSov2> i was freaking out
[2:49] <via> cool
[2:49] <TheSov2> it even let you virtual the chipset version, I dont know the purpose of it, maybe so you can install a central image without dealing with drivers but it was very cool
[2:50] <TheSov2> slow as heck to boot though
[2:56] * lifeboy (~roland@196.32.233.103) has joined #ceph
[3:04] <TheSov2> ok im gonna have to go ahead and try the hammer monitors with the firefly osd's
[3:04] <TheSov2> hopefully i dont waste a bunch of time on it for nothing
[3:06] * shyu (~Shanzhi@119.254.120.66) Quit (Remote host closed the connection)
[3:08] * shyu (~Shanzhi@119.254.120.66) has joined #ceph
[3:09] * lifeboy (~roland@196.32.233.103) Quit (Ping timeout: 480 seconds)
[3:36] * georgem (~Adium@23-91-150-96.cpe.pppoe.ca) has joined #ceph
[3:46] * debian112 (~bcolbert@24.126.201.64) Quit (Quit: Leaving.)
[3:59] * DV_ (~veillard@2001:41d0:a:f29f::1) Quit (Ping timeout: 480 seconds)
[4:05] * scuttle is now known as scuttle|afk
[4:09] * lifeboy (~roland@196.32.233.103) has joined #ceph
[4:10] * DV_ (~veillard@2001:41d0:1:d478::1) has joined #ceph
[4:11] * doppelgrau (~doppelgra@pd956d116.dip0.t-ipconnect.de) Quit (Quit: doppelgrau)
[4:16] * flisky (~Thunderbi@106.39.60.34) has joined #ceph
[4:20] * Concubidated (~Adium@192.170.161.35) Quit (Quit: Leaving.)
[4:28] * DV_ (~veillard@2001:41d0:1:d478::1) Quit (Ping timeout: 480 seconds)
[4:28] * DV (~veillard@2001:41d0:a:f29f::1) Quit (Ping timeout: 480 seconds)
[4:28] * kefu (~kefu@114.92.100.239) has joined #ceph
[4:29] * shang (~ShangWu@175.41.48.77) has joined #ceph
[4:31] * kefu (~kefu@114.92.100.239) Quit (Max SendQ exceeded)
[4:31] * jclm1 (~jclm@m822536d0.tmodns.net) has joined #ceph
[4:32] * bkopilov (~bkopilov@109.65.167.66) Quit (Ping timeout: 480 seconds)
[4:34] * kefu (~kefu@114.92.100.239) has joined #ceph
[4:36] * DV_ (~veillard@2001:41d0:a:f29f::1) has joined #ceph
[4:38] * DV (~veillard@2001:41d0:1:d478::1) has joined #ceph
[4:39] * jclm (~jclm@50-206-204-8-static.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[4:41] * zhaochao (~zhaochao@111.161.77.241) has joined #ceph
[4:42] * sankarshan (~sankarsha@183.87.39.242) has joined #ceph
[4:44] * Kupo1 (~tyler.wil@23.111.254.159) Quit (Read error: Connection reset by peer)
[5:01] * overclk (~overclk@121.244.87.117) has joined #ceph
[5:04] * snakamoto (~Adium@192.16.26.2) has joined #ceph
[5:14] <kefu> snakamoto: http://ceph.com/packages/ceph-extras/debian/dists/
[5:15] <kefu> so you are looking for ceph-deploy.
[5:15] <snakamoto> No, we are not using ceph-deploy
[5:18] <kefu> snakamoto: i walked in the http://ceph.com/packages/ceph-extras/debian/pool/, that's could be the most interesting package in that repo, though.
[5:18] <kefu> but i tend to agree with you, the ceph-extra repo does not have trusty in its distros.
[5:19] <kefu> but hammer does http://ceph.com/debian-hammer/dists/
[5:19] <snakamoto> This document states that Ceph Extras is required for installing Ceph: http://docs.ceph.com/docs/master/install/get-packages/#add-ceph-extras
[5:20] <snakamoto> It looks like I can install Ceph, even if I leave out the ceph-extras repo
[5:20] <kefu> yeah. you are right.
[5:21] <snakamoto> Is this safe to do? There isn't a ceph-specific version of libcurl or anything that needs to be installed?
[5:22] <kefu> snakamoto: i am not sure what is the secret sauce we put in the ceph-extra packages already cooked by the distros, like curl.
[5:23] * kefu looking into it.
[5:24] <kefu> snakamoto: before i find exactly what the "newer Ceph-enabled versions of packages " means, my wild guess is
[5:25] <kefu> some of the packages provided by ceph-extra are backports to the distro listed in http://ceph.com/packages/ceph-extras/debian/dists/ ,
[5:26] <kefu> and trusty is a relatively new release,
[5:27] <kefu> snakamoto: the packages used by ceph is already new enough to support ceph.
[5:28] <snakamoto> Ahhh, I see. If that's true, then ceph-extras is only necessary for older versions
[5:33] <kefu> snakamoto: i just checked the test suite of ceph, seems the ceph-extra repo is not added
[5:33] <kefu> s/test suite/test suites/
[5:34] <kefu> snakamoto: so i guess it's safe =D.
[5:34] <snakamoto> Great! Thank you so much for your help, Kefu!
[5:35] <kefu> snakamoto: with pleasure =)
[5:38] * Vacuum_ (~Vacuum@i59F7910C.versanet.de) has joined #ceph
[5:42] * OutOfNoWhere (~rpb@199.68.195.101) Quit (Ping timeout: 480 seconds)
[5:43] * snakamoto (~Adium@192.16.26.2) Quit (Quit: Leaving.)
[5:44] <ska> Do I need to start my osd id's at 0?
[5:45] <ska> I show a 0 evn though I don't have that section.
[5:45] * Vacuum__ (~Vacuum@88.130.201.240) Quit (Ping timeout: 480 seconds)
[5:47] * bkopilov (~bkopilov@nat-pool-tlv-t.redhat.com) has joined #ceph
[5:59] * georgem (~Adium@23-91-150-96.cpe.pppoe.ca) Quit (Quit: Leaving.)
[6:00] * sankarshan_ (~sankarsha@183.87.39.242) has joined #ceph
[6:00] * rotbeard (~redbeard@2a02:908:df10:d300:76f0:6dff:fe3b:994d) Quit (Ping timeout: 480 seconds)
[6:02] * sankarshan (~sankarsha@183.87.39.242) Quit (Ping timeout: 480 seconds)
[6:05] * jamespage (~jamespage@culvain.gromper.net) Quit (Quit: Coyote finally caught me)
[6:06] * jamespage (~jamespage@culvain.gromper.net) has joined #ceph
[6:19] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[6:29] * yguang11 (~yguang11@12.31.82.125) has joined #ceph
[6:31] * yguang11_ (~yguang11@nat-dip15.fw.corp.yahoo.com) has joined #ceph
[6:33] * rdas (~rdas@121.244.87.116) has joined #ceph
[6:34] * dgbaley27 (~matt@c-67-176-93-83.hsd1.co.comcast.net) has joined #ceph
[6:35] * derjohn_mob (~aj@x590ca836.dyn.telefonica.de) has joined #ceph
[6:35] * lifeboy (~roland@196.32.233.103) Quit (Ping timeout: 480 seconds)
[6:38] * yguang11 (~yguang11@12.31.82.125) Quit (Ping timeout: 480 seconds)
[6:47] * derjohn_mob (~aj@x590ca836.dyn.telefonica.de) Quit (Ping timeout: 480 seconds)
[6:51] * amote (~amote@121.244.87.116) has joined #ceph
[6:51] * zaitcev (~zaitcev@2001:558:6001:10:61d7:f51f:def8:4b0f) Quit (Quit: Bye)
[6:56] * karnan (~karnan@121.244.87.117) has joined #ceph
[7:01] * wschulze (~wschulze@cpe-69-206-240-164.nyc.res.rr.com) Quit (Quit: Leaving.)
[7:06] * trociny (~mgolub@93.183.239.2) has joined #ceph
[7:09] * jclm (~jclm@50-206-204-8-static.hfc.comcastbusiness.net) has joined #ceph
[7:15] * jclm1 (~jclm@m822536d0.tmodns.net) Quit (Ping timeout: 480 seconds)
[7:26] * sankarshan_ (~sankarsha@183.87.39.242) Quit (Ping timeout: 480 seconds)
[7:29] * sankarshan (~sankarsha@183.87.39.242) has joined #ceph
[7:31] * overclk (~overclk@121.244.87.117) Quit (Quit: Leaving)
[7:31] * overclk (~overclk@121.244.87.117) has joined #ceph
[7:40] * treenerd (~treenerd@83-64-142-10.zwischennetz.xdsl-line.inode.at) has joined #ceph
[7:40] * treenerd (~treenerd@83-64-142-10.zwischennetz.xdsl-line.inode.at) Quit ()
[7:43] * yguang11_ (~yguang11@nat-dip15.fw.corp.yahoo.com) Quit ()
[7:46] * shang (~ShangWu@175.41.48.77) Quit (Remote host closed the connection)
[7:46] * shohn (~shohn@dslb-088-075-235-250.088.075.pools.vodafone-ip.de) has joined #ceph
[7:48] * derjohn_mob (~aj@tmo-110-165.customers.d1-online.com) has joined #ceph
[7:51] * karnan (~karnan@121.244.87.117) Quit (Quit: Leaving)
[7:52] * karnan (~karnan@121.244.87.117) has joined #ceph
[7:58] * shang (~ShangWu@175.41.48.77) has joined #ceph
[8:04] * badone__ (~brad@CPE-121-215-241-179.static.qld.bigpond.net.au) Quit (Ping timeout: 480 seconds)
[8:05] * MACscr1 (~Adium@2601:247:4102:c3ac:5d73:b8e1:a763:499) has joined #ceph
[8:07] * badone (~brad@CPE-121-215-241-179.static.qld.bigpond.net.au) has joined #ceph
[8:10] * MACscr (~Adium@2601:247:4102:c3ac:8c86:b950:b3c8:723f) Quit (Ping timeout: 480 seconds)
[8:13] * oro (~oro@188-143-121-132.pool.digikabel.hu) has joined #ceph
[8:16] * derjohn_mob (~aj@tmo-110-165.customers.d1-online.com) Quit (Ping timeout: 480 seconds)
[8:17] * derjohn_mobi (~aj@tmo-107-136.customers.d1-online.com) has joined #ceph
[8:17] * Nacer (~Nacer@203-206-190-109.dsl.ovh.fr) has joined #ceph
[8:20] * Hemanth (~Hemanth@121.244.87.117) has joined #ceph
[8:28] * Sysadmin88 (~IceChat77@2.124.164.69) Quit (Quit: REALITY.SYS Corrupted: Re-boot universe? (Y/N/Q))
[8:29] * finster (~finster@2a01:4f8:d15:1000::2) has joined #ceph
[8:31] * Hemanth (~Hemanth@121.244.87.117) Quit (Quit: Leaving)
[8:31] * Hemanth (~Hemanth@121.244.87.117) has joined #ceph
[8:34] * finster (~finster@2a01:4f8:d15:1000::2) has left #ceph
[8:35] * derjohn_mobi (~aj@tmo-107-136.customers.d1-online.com) Quit (Ping timeout: 480 seconds)
[8:35] * badone (~brad@CPE-121-215-241-179.static.qld.bigpond.net.au) Quit (Ping timeout: 480 seconds)
[8:35] * dugravot6 (~dugravot6@nat-persul-montet.wifi.univ-lorraine.fr) has joined #ceph
[8:36] * Nacer (~Nacer@203-206-190-109.dsl.ovh.fr) Quit (Remote host closed the connection)
[8:38] * kefu (~kefu@114.92.100.239) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[8:44] * lifeboy (~roland@196.45.30.156) has joined #ceph
[8:46] * MACscr1 (~Adium@2601:247:4102:c3ac:5d73:b8e1:a763:499) Quit (Quit: Leaving.)
[8:46] * T1w (~jens@node3.survey-it.dk) has joined #ceph
[8:48] * MACscr (~Adium@2601:247:4102:c3ac:bc8e:55dc:2513:f7ce) has joined #ceph
[8:53] * topro (~prousa@host-62-245-142-50.customer.m-online.net) Quit (Ping timeout: 480 seconds)
[8:53] * Be-El (~quassel@fb08-bcf-pc01.computational.bio.uni-giessen.de) has joined #ceph
[8:53] <Be-El> hi
[8:56] * topro (~prousa@host-62-245-142-50.customer.m-online.net) has joined #ceph
[9:01] * Concubidated (~Adium@23.91.33.7) has joined #ceph
[9:01] * shylesh__ (~shylesh@121.244.87.124) has joined #ceph
[9:03] * macjack (~macjack@122.146.93.152) Quit (Remote host closed the connection)
[9:05] * kawa2014 (~kawa@89.184.114.246) has joined #ceph
[9:06] * derjohn_mob (~aj@fw.gkh-setu.de) has joined #ceph
[9:08] * bkopilov (~bkopilov@nat-pool-tlv-t.redhat.com) Quit (Ping timeout: 480 seconds)
[9:11] * kefu (~kefu@114.92.100.239) has joined #ceph
[9:11] * analbeard (~shw@support.memset.com) has joined #ceph
[9:12] * topro (~prousa@host-62-245-142-50.customer.m-online.net) Quit (Ping timeout: 480 seconds)
[9:16] * lifeboy (~roland@196.45.30.156) Quit (Ping timeout: 480 seconds)
[9:17] * dgurtner (~dgurtner@178.197.231.10) has joined #ceph
[9:18] * wicope (~wicope@0001fd8a.user.oftc.net) has joined #ceph
[9:20] * bkopilov (~bkopilov@nat-pool-tlv-u.redhat.com) has joined #ceph
[9:23] * oro (~oro@188-143-121-132.pool.digikabel.hu) Quit (Ping timeout: 480 seconds)
[9:26] * bitserker (~toni@88.87.194.130) has joined #ceph
[9:30] * wesbell_ (~wesbell@vpngac.ccur.com) Quit (Read error: Connection reset by peer)
[9:31] * wesbell_ (~wesbell@vpngac.ccur.com) has joined #ceph
[9:31] * thomnico (~thomnico@bob13-2-88-172-8-244.fbx.proxad.net) has joined #ceph
[9:36] * gaveen (~gaveen@123.231.123.35) has joined #ceph
[9:40] * derjohn_mob (~aj@fw.gkh-setu.de) Quit (Ping timeout: 480 seconds)
[9:42] * dgbaley27 (~matt@c-67-176-93-83.hsd1.co.comcast.net) Quit (Ping timeout: 480 seconds)
[9:46] * Nacer (~Nacer@252-87-190-213.intermediasud.com) has joined #ceph
[9:47] * fridim_ (~fridim@56-198-190-109.dsl.ovh.fr) has joined #ceph
[9:49] * xarses (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) Quit (Remote host closed the connection)
[9:50] * xarses (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) has joined #ceph
[9:50] * derjohn_mob (~aj@fw.gkh-setu.de) has joined #ceph
[9:50] * MACscr (~Adium@2601:247:4102:c3ac:bc8e:55dc:2513:f7ce) Quit (Read error: Connection reset by peer)
[9:50] * MACscr (~Adium@2601:247:4102:c3ac:bc8e:55dc:2513:f7ce) has joined #ceph
[9:53] * thomnico (~thomnico@bob13-2-88-172-8-244.fbx.proxad.net) Quit (Ping timeout: 480 seconds)
[9:54] * caban (~unknown@office.siplabs.ru) has joined #ceph
[9:55] <caban> Hi all, Q: swift -A http://ip/auth -U testuser:swift -K fwXJGiPCq1Z9ugXlbjAjqPfJjw7bA71iicVIjt list A: Auth GET failed: 403 Forbidden
[9:55] <caban> "swift_keys": [
[9:55] <caban> { "user": "testuser:swift",
[9:55] <caban> "secret_key": "fwXJGiPCq1Z9ugXlbjAjqPfJjw7bA71iicVIjtE"}],
[9:55] <caban> why I always get 403 on auth?
[9:55] * topro (~prousa@host-62-245-142-50.customer.m-online.net) has joined #ceph
[10:00] * topro (~prousa@host-62-245-142-50.customer.m-online.net) Quit (Read error: Connection reset by peer)
[10:02] * mattch (~mattch@pcw3047.see.ed.ac.uk) has joined #ceph
[10:02] * fam is now known as fam_away
[10:03] * fam_away is now known as fam
[10:05] * topro (~prousa@host-62-245-142-50.customer.m-online.net) has joined #ceph
[10:07] * thomnico (~thomnico@cro38-2-88-180-16-18.fbx.proxad.net) has joined #ceph
[10:07] <topro> hi there, just having some big troubles with cephfs after upgrading to giant, firefly was running rather stable since its release
[10:07] * garphy`aw is now known as garphy
[10:07] <topro> rather standard setup, three nodes, 8 OSDs each, one MDS and MON each
[10:07] <topro> manual setup, no ceph deploy
[10:08] <topro> ceph.conf containing no special options, just minimum set of settings like hostnames and ip addresses
[10:09] <topro> only additional setting is custom journal location + journal size (2048)
[10:09] * fam is now known as fam_away
[10:09] <topro> running about 20 clients using cephfs as home dir
[10:09] <topro> first strange thing i noticed after giant upgrade was "client failed to respond to cache pressure"
[10:10] <topro> then MDS started to randomly switch from one active to standby ones, giving a minute of "no service" each time
[10:10] * TMM (~hp@sams-office-nat.tomtomgroup.com) has joined #ceph
[10:11] <topro> trying to stop MDS to switch to standby-ones I stopped standby MDS instances, then things started to go worse
[10:11] <topro> MDS tried to switch no longer running standby, then back, to last remaining running MDS instance
[10:12] <topro> this MDS process memory footprint then grew to make the machine swap massively and MDS logfile on that host grew to some 30 GB until /var partition was filled up to 100%
[10:14] * todin (tuxadero@kudu.in-berlin.de) has joined #ceph
[10:15] <topro> the huge logfile seems to contain huge ammounts of binary data
[10:15] * fam_away is now known as fam
[10:17] * lifeboy (~roland@105.10.201.144) has joined #ceph
[10:19] * fam is now known as fam_away
[10:20] * fam_away is now known as fam
[10:25] * fam is now known as fam_away
[10:28] * lifeboy (~roland@105.10.201.144) Quit (Ping timeout: 480 seconds)
[10:29] * branto (~branto@nat-pool-brq-t.redhat.com) has joined #ceph
[10:30] * nardial (~ls@dslb-178-011-179-229.178.011.pools.vodafone-ip.de) has joined #ceph
[10:32] * fam_away is now known as fam
[10:35] * doppelgrau (~doppelgra@pd956d116.dip0.t-ipconnect.de) has joined #ceph
[10:47] * nsoffer (~nsoffer@109.226.17.231) has joined #ceph
[10:52] * vikhyat (~vumrao@121.244.87.116) has joined #ceph
[10:54] * topro_ (~prousa@host-62-245-142-50.customer.m-online.net) has joined #ceph
[10:55] * topro_ is now known as topro__
[10:56] * garphy is now known as garphy`aw
[10:57] * kefu (~kefu@114.92.100.239) Quit (Max SendQ exceeded)
[10:58] * kefu (~kefu@114.92.100.239) has joined #ceph
[10:59] * zhaochao (~zhaochao@111.161.77.241) Quit (Quit: ChatZilla 0.9.91.1 [Iceweasel 38.0.1/20150526223604])
[11:03] * garphy`aw is now known as garphy
[11:06] * dugravot6 (~dugravot6@nat-persul-montet.wifi.univ-lorraine.fr) Quit (Quit: Leaving.)
[11:11] * Debesis (~0x@143.252.117.89.static.mezon.lt) has joined #ceph
[11:20] * bkopilov (~bkopilov@nat-pool-tlv-u.redhat.com) Quit (Ping timeout: 480 seconds)
[11:25] * mancdaz (~mancdaz@2a00:1a48:7806:117:be76:4eff:fe08:7623) Quit (Quit: ZNC - http://znc.in)
[11:25] * mancdaz (~mancdaz@2a00:1a48:7806:117:be76:4eff:fe08:7623) has joined #ceph
[11:28] * bkopilov (~bkopilov@nat-pool-tlv-t.redhat.com) has joined #ceph
[11:34] * DV (~veillard@2001:41d0:1:d478::1) Quit (Ping timeout: 480 seconds)
[11:35] * DV_ (~veillard@2001:41d0:a:f29f::1) Quit (Ping timeout: 480 seconds)
[11:40] * kefu (~kefu@114.92.100.239) Quit (Max SendQ exceeded)
[11:41] * badone (~brad@CPE-121-215-241-179.static.qld.bigpond.net.au) has joined #ceph
[11:41] * kefu (~kefu@114.92.100.239) has joined #ceph
[11:44] * DV (~veillard@2001:41d0:a:f29f::1) has joined #ceph
[11:45] * DV_ (~veillard@2001:41d0:a:f29f::1) has joined #ceph
[11:47] * nsoffer (~nsoffer@109.226.17.231) Quit (Ping timeout: 480 seconds)
[11:49] * linjan (~linjan@195.110.41.9) has joined #ceph
[11:54] * kefu (~kefu@114.92.100.239) Quit (Max SendQ exceeded)
[11:55] * koollman_ is now known as koollman
[11:55] * kefu (~kefu@114.92.100.239) has joined #ceph
[11:57] <jcsp1> toppro: any particular reason you're using giant instead of hammer? giant isn't an LTS release
[11:57] <jcsp1> the "client failed to respond to cache pressure" is likely a pre-existing issue, the health check for detecting it is just new in giant
[11:58] <jcsp1> (usually people get that with older kernel clients)
[11:58] <jcsp1> ^topro
[11:59] <jcsp1> you may want to turn the MDS debug settings down to get a more manageable log, even on lower settings it should be telling you why it's stopping
[12:08] * dugravot6 (~dugravot6@nat-persul-montet.wifi.univ-lorraine.fr) has joined #ceph
[12:13] * oro (~oro@adsl-89-132-24-43.monradsl.monornet.hu) has joined #ceph
[12:22] * oro (~oro@adsl-89-132-24-43.monradsl.monornet.hu) Quit (Ping timeout: 480 seconds)
[12:26] <Be-El> if you want to use an EC pool in cepfs, does it need its own cache tier, or can you share a cache tier with other pools?
[12:27] * thomnico (~thomnico@cro38-2-88-180-16-18.fbx.proxad.net) Quit (Ping timeout: 480 seconds)
[12:29] <jcsp1> I have trouble remembering the rules about 1:N and N:1 cache tier relationships. Try it :-)
[12:30] * nhm (~nhm@184-97-242-33.mpls.qwest.net) Quit (Ping timeout: 480 seconds)
[12:31] <gregsfortytwo> I think you can have multiple cache pools for a single backing pool, if they're read-only caches
[12:31] <gregsfortytwo> but I don't think you can pull multiple backing pools into a single cache pool
[12:34] <Be-El> so it's safer to create a separate cache pool. the other pool is for rbd used by production services, so it's not good if i loose the cache...
[12:34] * arbrandes (~arbrandes@189.78.72.151) has joined #ceph
[12:37] * kefu (~kefu@114.92.100.239) Quit (Max SendQ exceeded)
[12:39] * kefu (~kefu@114.92.100.239) has joined #ceph
[12:40] * thomnico (~thomnico@bob13-2-88-172-8-244.fbx.proxad.net) has joined #ceph
[12:41] * flisky (~Thunderbi@106.39.60.34) Quit (Quit: flisky)
[12:44] <topro__> jcsp1: sorry, that was a typo here. I upgraded to hammer of course
[12:46] * karnan (~karnan@121.244.87.117) Quit (Remote host closed the connection)
[12:46] <topro__> jcsp1: using kernel client of latest linux 3.16 from debian wheezy backports. so it gets some of the canonical backports of later kernels. don't know exactly if this covers ceph kernel client stuff being backported
[12:46] * dugravot6 (~dugravot6@nat-persul-montet.wifi.univ-lorraine.fr) Quit (Ping timeout: 480 seconds)
[12:47] <topro__> jcsp1: mds runs with default log/debug settings, would that include binary blobs in logfiles, as that is what it looks like
[12:48] <Be-El> how do you add an EC pool to cephfs? ceph mds add_data_pool complains about the pool being an EC pool
[12:50] <topro__> the biggest problems I am encountering right now is random MDS host switching (active hopping from one host to a standby host about ance a day) and MDS memory footprint growing indefinitely from time to time also creating binary-stuff filled huge logfiles at those times
[12:51] * shyu (~Shanzhi@119.254.120.66) Quit (Remote host closed the connection)
[12:51] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit (Ping timeout: 480 seconds)
[12:58] * DV (~veillard@2001:41d0:a:f29f::1) Quit (Ping timeout: 480 seconds)
[12:58] * DV_ (~veillard@2001:41d0:a:f29f::1) Quit (Ping timeout: 480 seconds)
[12:58] * DV (~veillard@2001:41d0:1:d478::1) has joined #ceph
[13:03] * lx0 (~aoliva@lxo.user.oftc.net) has joined #ceph
[13:08] * DV_ (~veillard@2001:41d0:1:d478::1) has joined #ceph
[13:09] * ira (~ira@c-71-233-225-22.hsd1.ma.comcast.net) has joined #ceph
[13:10] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[13:10] <jcsp1> topro_: the switching you're seeing is normal behaviour when an MDS crashes or becomes unresponsive
[13:11] <jcsp1> so the question is why is it crashing
[13:11] <jcsp1> as for binary blobs in log files, that's not usual
[13:11] <jcsp1> see what's at the start of such a line
[13:13] * fdmanana__ (~fdmanana@bl14-141-55.dsl.telepac.pt) Quit (Ping timeout: 480 seconds)
[13:14] * thomnico_ (~thomnico@cro38-2-88-180-16-18.fbx.proxad.net) has joined #ceph
[13:15] * amote (~amote@121.244.87.116) Quit (Quit: Leaving)
[13:16] * nsoffer (~nsoffer@nat-pool-tlv-t.redhat.com) has joined #ceph
[13:16] * DV (~veillard@2001:41d0:1:d478::1) Quit (Ping timeout: 480 seconds)
[13:20] * thomnico (~thomnico@bob13-2-88-172-8-244.fbx.proxad.net) Quit (Ping timeout: 480 seconds)
[13:20] <topro__> jcsp1: I've been so dumb to delete those big logfiles as it was really hard to handle some 30G file. when (if) I get a new one, I'll have a look
[13:20] <topro__> I assume that the growing memory footprint (bug?!?) is directly related to MDS writing binary nonsense to logfile, maybe?
[13:21] * fdmanana__ (~fdmanana@bl14-141-55.dsl.telepac.pt) has joined #ceph
[13:23] <topro__> could that all be related to maybe linux 3.16 kernel client missing some features (see mds helth report about cache pressure). Somwhere on the ML i read from greg or sage that this messages doesn't mean any harm other than a slightly increased memory consumption of the MDS.
[13:23] <topro__> I hope what I see is not what he meant with slightly increased, though :P
[13:23] * cdelatte (~cdelatte@cpe-172-72-105-98.carolina.res.rr.com) Quit (Quit: This computer has gone to sleep)
[13:27] * DV (~veillard@2001:41d0:a:f29f::1) has joined #ceph
[13:29] * oro (~oro@apn-94-44-254-175.vodafone.hu) has joined #ceph
[13:33] * ganders (~root@190.2.42.21) has joined #ceph
[13:34] * fdmanana__ (~fdmanana@bl14-141-55.dsl.telepac.pt) Quit (Ping timeout: 480 seconds)
[13:36] * rwheeler (~rwheeler@pool-173-48-214-9.bstnma.fios.verizon.net) has joined #ceph
[13:37] * MACscr1 (~Adium@c-98-214-160-70.hsd1.il.comcast.net) has joined #ceph
[13:39] * i_m (~ivan.miro@deibp9eh1--blueice4n1.emea.ibm.com) has joined #ceph
[13:42] * dneary (~dneary@pool-96-252-45-212.bstnma.fios.verizon.net) has joined #ceph
[13:44] * shang (~ShangWu@175.41.48.77) Quit (Quit: Ex-Chat)
[13:45] * MACscr (~Adium@2601:247:4102:c3ac:bc8e:55dc:2513:f7ce) Quit (Ping timeout: 480 seconds)
[13:50] * cdelatte (~cdelatte@204-235-114.65.twcable.com) has joined #ceph
[13:50] * kanagaraj (~kanagaraj@ip-64-134-240-34.public.wayport.net) has joined #ceph
[13:51] * kefu (~kefu@114.92.100.239) Quit (Max SendQ exceeded)
[13:52] * kefu (~kefu@114.92.100.239) has joined #ceph
[13:53] * SkyEye (~gaveen@123.231.112.221) has joined #ceph
[13:54] * linjan (~linjan@195.110.41.9) Quit (Ping timeout: 480 seconds)
[13:55] <jcsp1> when starting an MDS, growing memory consumption is what you expect as it replays the journal and receives replayed requests from the clients. The odd part would be if it's growing larger than it was before the upgrade.
[13:55] <jcsp1> given that your clients are failing to release capabilities, you may find that your MDS is happier starting once they're unmounted.
[13:56] * sankarshan (~sankarsha@183.87.39.242) Quit (Read error: Connection reset by peer)
[13:59] * gaveen (~gaveen@123.231.123.35) Quit (Ping timeout: 480 seconds)
[14:01] * marrusl_ (~mark@cpe-67-247-9-253.nyc.res.rr.com) has joined #ceph
[14:01] * marrusl_ (~mark@cpe-67-247-9-253.nyc.res.rr.com) Quit ()
[14:03] * linjan (~linjan@46.210.122.18) has joined #ceph
[14:04] <topro__> jcsp1: on startup it is growing like the same as it did before upgrade (1GB ~ 1.5GB). but then after some time of operation it suddenly starts to grow to ten times that size. could not yet figure out what triggers that, though
[14:04] <topro__> but its the same moment when it would start writing those binary stuff to logfile as well
[14:08] * dugravot6 (~dugravot6@nat-persul-plg.wifi.univ-lorraine.fr) has joined #ceph
[14:09] * kefu (~kefu@114.92.100.239) Quit (Max SendQ exceeded)
[14:10] * kefu (~kefu@li413-226.members.linode.com) has joined #ceph
[14:10] * sugoruyo (~georgev@paarthurnax.esc.rl.ac.uk) has joined #ceph
[14:11] * sankarshan (~sankarsha@183.87.39.242) has joined #ceph
[14:11] <sugoruyo> hi folks, anyone come across a problem where `rados -p <erasure-pool> ls` hangs and times out?
[14:12] <sugoruyo> I have a specific EC pool that just won't respond to any object operation
[14:14] * jdillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) has joined #ceph
[14:14] * t0rn (~ssullivan@2607:fad0:32:a02:56ee:75ff:fe48:3bd3) has joined #ceph
[14:15] * dneary (~dneary@pool-96-252-45-212.bstnma.fios.verizon.net) Quit (Ping timeout: 480 seconds)
[14:16] * badone (~brad@CPE-121-215-241-179.static.qld.bigpond.net.au) Quit (Ping timeout: 480 seconds)
[14:21] * oro (~oro@apn-94-44-254-175.vodafone.hu) Quit (Ping timeout: 480 seconds)
[14:26] * overclk (~overclk@121.244.87.117) Quit (Quit: Leaving)
[14:27] * DV (~veillard@2001:41d0:a:f29f::1) Quit (Ping timeout: 480 seconds)
[14:29] * wschulze (~wschulze@cpe-69-206-240-164.nyc.res.rr.com) has joined #ceph
[14:29] <zenpac> How do I bring an OSD up?
[14:31] <zenpac> http://paste.debian.net/255982/ shows my "ceph osd dump" but all oSD's are down.
[14:35] <sugoruyo> zenpac: are your OSD daemons running?
[14:35] <zenpac> I manually put them in the "in" state, but I think that was a mistake...
[14:35] <zenpac> Yes..
[14:35] <zenpac> /usr/bin/python /usr/bin/ceph --cluster=ceph --name=mon. --keyring=/var/lib/ceph/mon/ceph-0/keyring auth get-or-create client.admin mon allow * osd allow * mds allow
[14:36] * daviddcc (~dcasier@84.197.151.77.rev.sfr.net) Quit (Ping timeout: 480 seconds)
[14:36] * DV (~veillard@2001:41d0:a:f29f::1) has joined #ceph
[14:37] * MVenesio (~MVenesio@186.136.37.21) has joined #ceph
[14:37] <zenpac> It seems like i'm close.. but just short of working..
[14:39] <sugoruyo> zenpac: maybe take a look at OSDs' logs, it could tell you what's wrong on their end
[14:39] * bkopilov (~bkopilov@nat-pool-tlv-t.redhat.com) Quit (Ping timeout: 480 seconds)
[14:39] * MVenesio (~MVenesio@186.136.37.21) Quit ()
[14:39] <zenpac> http://paste.debian.net/255983/
[14:40] * amote (~amote@121.244.87.116) has joined #ceph
[14:40] <zenpac> Thats my crush dump
[14:40] <zenpac> ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-0: (2) No such file or directory
[14:42] <zenpac> http://paste.debian.net/256060/ is the log and content of the error folder..
[14:42] <sugoruyo> does the directory exist? is a filesystem mounted under it?
[14:44] <zenpac> http://paste.debian.net/256112/
[14:45] <sugoruyo> looks like you're missing some stuff there
[14:46] <sugoruyo> for instance, mine has a file called superblock in it... are you sure it's been initialised properly as an OSD?
[14:46] <zenpac> Haha! I wish I knew.
[14:47] <zenpac> I may have thrown the keyring in there too soon.
[14:47] <zenpac> I copied the keyring in before I initialized the osd..
[14:47] * geoffr (~geoffr@vpn-router1.dur1.host-h.net) has joined #ceph
[14:47] <zenpac> Perhaps that is my mistake.
[14:48] * kanagaraj (~kanagaraj@ip-64-134-240-34.public.wayport.net) Quit (Ping timeout: 480 seconds)
[14:48] <geoffr> Hi all, does anyone perhaps have a link for the changelog /release notes for ceph version 0.80.10 ? My Debian based boxes are wanting to upgrade to this. The last one I can find is http://ceph.com/docs/master/_downloads/v0.80.9.txt
[14:49] * dugravot61 (~dugravot6@dn-infra-04.lionnois.univ-lorraine.fr) has joined #ceph
[14:49] <Be-El> zenpac: just stop doing it manually and use ceph-disk prepare for setting up the osds....
[14:49] * dugravot61 (~dugravot6@dn-infra-04.lionnois.univ-lorraine.fr) Quit ()
[14:50] * dugravot61 (~dugravot6@dn-infra-04.lionnois.univ-lorraine.fr) has joined #ceph
[14:50] <sugoruyo> zenpac: Be-El's advice should be heeded, use the provided deployment tools or life can get painful...
[14:51] <Be-El> sugoruyo: doing it manually is ok, but you should first understand how it actually works (and what the deployment tools really do)
[14:52] * dugravot6 (~dugravot6@nat-persul-plg.wifi.univ-lorraine.fr) Quit (Ping timeout: 480 seconds)
[14:53] <zenpac> Yes, but I need to understand how it works in a deep way..
[14:53] <zenpac> I may end up using those tools though.
[14:54] <Be-El> zenpac: so use the tools first, get a running setup, try to reproduce manually afterwards
[14:54] * dgbaley27 (~matt@c-67-176-93-83.hsd1.co.comcast.net) has joined #ceph
[14:54] * georgem (~Adium@fwnat.oicr.on.ca) has joined #ceph
[14:54] <Be-El> zenpac: if you don't know how a valid osd directory looks like, you will never be able to create it manually
[14:55] * daviddcc (~dcasier@80.215.195.3) has joined #ceph
[14:55] <zenpac> I am using ceph-disk-prepare..
[14:55] <zenpac> yes.. I am ..
[14:55] <zenpac> I'll show
[14:55] * linjan (~linjan@46.210.122.18) Quit (Ping timeout: 480 seconds)
[14:57] * brad_mssw (~brad@66.129.88.50) has joined #ceph
[14:59] <zenpac> Be-El: If I use it manually, can i: mkfs /dev/sda2, mount /dev/sda2, ceph-disk-prepare .... ?
[15:00] * DV (~veillard@2001:41d0:a:f29f::1) Quit (Ping timeout: 480 seconds)
[15:00] <Be-El> zenpac: use just ceph-disk prepare without any other command....
[15:00] <Be-El> zenpac: it will create the filesystem, journal etc.
[15:00] <zenpac> http://paste.debian.net/256136/
[15:00] <Be-El> zenpac: it will even wire the osd to udev to start it automatically at boot (if supported)
[15:01] <zenpac> Do I specify the partition, or the directory?
[15:01] <Be-El> the partition or the whole block device if you want to use the whole disk
[15:02] * t0rn (~ssullivan@2607:fad0:32:a02:56ee:75ff:fe48:3bd3) has left #ceph
[15:02] <zenpac> Should it be mounted? I use /dev/sda3 as journal paritition and /dev/sda2 as data.
[15:02] * linjan (~linjan@46.210.122.18) has joined #ceph
[15:02] <Be-El> zenpac: no. do not mount anything
[15:04] <zenpac> http://paste.debian.net/256148/
[15:05] <zenpac> Do I need to specify my config file path?
[15:06] * primechuck (~primechuc@host-95-2-129.infobunker.com) has joined #ceph
[15:06] <Be-El> if the config is not in the default location, yes
[15:06] <zenpac> Its in default.. /etc/ceph/ceph.conf
[15:06] <Be-El> you can start the osd with 'ceph-disk activate /dev/sda2'
[15:07] <zenpac> Do I need to mount it, or copy the keyring into it?
[15:07] <Be-El> nope
[15:07] <zenpac> I do have an entry in /etc/fstab for it though.. Should I leave that?
[15:07] * lifeboy (~roland@196.45.30.156) has joined #ceph
[15:07] <zenpac> /dev/sda2 /var/lib/ceph/osd/ceph-0 xfs rw,noexec,nodev,noatime,nodiratime,barrier=0 0 0
[15:08] <Be-El> leave it for now. there is a better way to handle osd mounts
[15:08] * rdas (~rdas@121.244.87.116) Quit (Quit: Leaving)
[15:08] <zenpac> ok.. thank you.... very much..
[15:08] <Be-El> you're welcome
[15:08] <Be-El> do you use gpt partition table for /dev/sda?
[15:09] <zenpac> Unfortunately no..
[15:10] <zenpac> Its a dos disklabel on Vsphere vm.
[15:10] <Be-El> in that case you need to manually mount the osd partitions and start the osd
[15:10] * DV (~veillard@2001:41d0:1:d478::1) has joined #ceph
[15:10] <zenpac> Next time I'll build with GPT.
[15:11] * kefu (~kefu@li413-226.members.linode.com) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[15:11] <Be-El> if you would use gpt partition tables, 'ceph-disk prepare' assigns a partition type uuid that is recognized by a udev rule provided by ceph -> as soon as the kernel scans the partiion table, the osd is started automatically
[15:11] <Be-El> that's why ceph-disk creates gpt partition tables by default
[15:11] <zenpac> Thats cool.. I run debian 8, btw..
[15:12] <Be-El> if you really want to manually create osd, have a look at the source code of ceph-disk. it's just a python script
[15:12] <zenpac> It sounds like next time: i need to have a separate disk for Ceph... not a parittion on my / disk.
[15:12] <Be-El> i cannot comment on the state of systemd support for debian in the ceph package, i haven't used it yet
[15:13] <zenpac> Be-El: I could not activate: http://paste.debian.net/256151/
[15:14] <zenpac> Its looking for fsid.
[15:14] <Be-El> do you have a fsid entry in your ceph configuration?
[15:14] <zenpac> yes..
[15:14] <zenpac> But ceph isn't running.. Should mon be running?
[15:15] <zenpac> I shut it all down .
[15:15] <Be-El> no, the fsid should have been copied into the osd directory by ceph-disk prepare
[15:15] <Be-El> start the mon and retry setting up the osd with ceph-disk prepare
[15:16] <zenpac> ok.. Yea the fsid it has it all 0's
[15:21] * kanagaraj (~kanagaraj@nat-pool-bos-t.redhat.com) has joined #ceph
[15:21] <zenpac> ceph_fsid is all 0, the fsid there is not the one specified in the /etc/ceph/ceph.conf
[15:22] <zenpac> I only specify one fsid in the ceph.conf in the global area.
[15:22] <Be-El> that's right, and that fsid should be written to the fsid file in the osd directory by ceph-disk
[15:23] <Be-El> so remove the osd again and recreate it with ceph-disk prepare, with running mons this time
[15:23] <zenpac> ok.. perhaps I needd to specify --cluster ceph in my ceph-disk prepare.
[15:23] <zenpac> Thats how I just did it.
[15:24] <Be-El> you can also try to specify it on the command line, but as far as i remember ceph disk falls back to get the fsid from the configuration file if none is given
[15:26] * yanzheng (~zhyan@182.139.20.134) Quit (Quit: This computer has gone to sleep)
[15:26] <zenpac> Its liking my --cluster ceph.. But its missing the keyring.. I need to copy it to the directory I thinks.
[15:26] * yanzheng (~zhyan@182.139.20.134) has joined #ceph
[15:27] * tupper (~tcole@2001:420:2280:1272:8900:f9b8:3b49:567e) has joined #ceph
[15:27] <Be-El> no, this should be done automatically. something else is not correct in your setup
[15:27] <zenpac> http://paste.debian.net/256162/ error is now moved to authentication.
[15:27] <zenpac> my keyring is /etc/ceph/keyring
[15:28] <zenpac> Its listed in ceph.conf too in the global section.
[15:28] <Be-El> do you have the admin key in the keyring file?
[15:30] * squizzi (~squizzi@2602:306:bc59:85f0:3ea9:f4ff:fe5a:6064) has joined #ceph
[15:30] <zenpac> I think so..
[15:30] <zenpac> client.admin
[15:30] <zenpac> allows * on all mon,mds,osd
[15:31] <Be-El> you also need the osd bootstrap key. but i do not remember how to create it and where to put it
[15:34] <zenpac> ok..
[15:34] * dgbaley27 (~matt@c-67-176-93-83.hsd1.co.comcast.net) Quit (Quit: Leaving.)
[15:35] * topro (~prousa@host-62-245-142-50.customer.m-online.net) Quit (Ping timeout: 480 seconds)
[15:36] * dlan (~dennis@116.228.88.131) Quit (Remote host closed the connection)
[15:36] * dlan (~dennis@116.228.88.131) has joined #ceph
[15:37] * topro (~prousa@host-62-245-142-50.customer.m-online.net) has joined #ceph
[15:37] * geoffr (~geoffr@vpn-router1.dur1.host-h.net) Quit (Quit: Leaving)
[15:40] * thomnico_ (~thomnico@cro38-2-88-180-16-18.fbx.proxad.net) Quit (Quit: Ex-Chat)
[15:40] * rlrevell (~leer@vbo1.inmotionhosting.com) has joined #ceph
[15:42] * nardial (~ls@dslb-178-011-179-229.178.011.pools.vodafone-ip.de) Quit (Quit: Leaving)
[15:44] * vikhyat (~vumrao@121.244.87.116) Quit (Quit: Leaving)
[15:44] * kefu (~kefu@114.92.100.239) has joined #ceph
[15:48] * daviddcc (~dcasier@80.215.195.3) Quit (Ping timeout: 480 seconds)
[15:53] * thomnico (~thomnico@cro38-2-88-180-16-18.fbx.proxad.net) has joined #ceph
[15:53] * kawa2014 (~kawa@89.184.114.246) Quit (Ping timeout: 480 seconds)
[15:54] * kawa2014 (~kawa@212.110.41.244) has joined #ceph
[15:54] * lifeboy (~roland@196.45.30.156) Quit (Quit: Ex-Chat)
[15:56] * jclm1 (~jclm@172.56.23.106) has joined #ceph
[15:57] * harold (~hamiller@71-94-227-123.dhcp.mdfd.or.charter.com) has joined #ceph
[15:58] * daviddcc (~dcasier@84.197.151.77.rev.sfr.net) has joined #ceph
[15:58] * T1w (~jens@node3.survey-it.dk) Quit (Ping timeout: 480 seconds)
[15:59] * kefu (~kefu@114.92.100.239) Quit (Max SendQ exceeded)
[16:00] * kefu (~kefu@114.92.100.239) has joined #ceph
[16:00] * jclm (~jclm@50-206-204-8-static.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[16:02] * dyasny (~dyasny@104.158.35.152) has joined #ceph
[16:02] * bkopilov (~bkopilov@bzq-109-65-167-66.red.bezeqint.net) has joined #ceph
[16:04] * capri (~capri@p4FC562D9.dip0.t-ipconnect.de) has joined #ceph
[16:06] * kefu is now known as kefu|afk
[16:07] * vikhyat (~vumrao@114.143.171.132) has joined #ceph
[16:08] * squizzi (~squizzi@2602:306:bc59:85f0:3ea9:f4ff:fe5a:6064) Quit (Quit: to the moon!)
[16:10] * squizzi (~squizzi@2602:306:bc59:85f0:3ea9:f4ff:fe5a:6064) has joined #ceph
[16:14] * kefu|afk (~kefu@114.92.100.239) Quit (Max SendQ exceeded)
[16:15] * kefu (~kefu@114.92.100.239) has joined #ceph
[16:18] * harold (~hamiller@71-94-227-123.dhcp.mdfd.or.charter.com) Quit (Quit: Leaving)
[16:25] * capri (~capri@p4FC562D9.dip0.t-ipconnect.de) Quit (Quit: Leaving)
[16:27] * vikhyat (~vumrao@114.143.171.132) Quit (Quit: Leaving)
[16:28] * xarses (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[16:33] * yanzheng (~zhyan@182.139.20.134) Quit (Quit: This computer has gone to sleep)
[16:36] <zenpac> Be-El: I've rebuilt: I still get problem with activation: http://paste.debian.net/256349/
[16:40] * thomnico (~thomnico@cro38-2-88-180-16-18.fbx.proxad.net) Quit (Read error: No route to host)
[16:40] * dugravot61 (~dugravot6@dn-infra-04.lionnois.univ-lorraine.fr) Quit (Quit: Leaving.)
[16:42] <zenpac> I was losely following this recipe: https://github.com/gc3-uzh-ch/ansible-playbooks/blob/master/roles/ceph/tasks/osd.yml
[16:43] * xarses (~xarses@12.164.168.117) has joined #ceph
[16:45] * nsoffer (~nsoffer@nat-pool-tlv-t.redhat.com) Quit (Ping timeout: 480 seconds)
[16:45] * Concubidated (~Adium@23.91.33.7) Quit (Quit: Leaving.)
[16:46] * nsoffer (~nsoffer@nat-pool-tlv-t.redhat.com) has joined #ceph
[16:46] * thomnico (~thomnico@cro38-2-88-180-16-18.fbx.proxad.net) has joined #ceph
[16:46] * debian112 (~bcolbert@24.126.201.64) has joined #ceph
[16:46] * sleinen (~Adium@2001:620:0:82::102) has joined #ceph
[16:47] * kawa2014 (~kawa@212.110.41.244) Quit (Ping timeout: 480 seconds)
[16:48] * kawa2014 (~kawa@89.184.114.246) has joined #ceph
[16:53] * linjan (~linjan@46.210.122.18) Quit (Ping timeout: 480 seconds)
[16:53] * fdmanana__ (~fdmanana@bl13-155-180.dsl.telepac.pt) has joined #ceph
[16:55] <zenpac> I'm clearly having issues with keyrings. I only have a single keyring file that is copied onto various locations.
[16:57] * RomeroJnr (~h0m3r@hosd.leaseweb.net) has joined #ceph
[16:57] * doppelgrau (~doppelgra@pd956d116.dip0.t-ipconnect.de) Quit (Read error: Connection reset by peer)
[16:57] * doppelgrau (~doppelgra@pd956d116.dip0.t-ipconnect.de) has joined #ceph
[16:58] * branto (~branto@nat-pool-brq-t.redhat.com) Quit (Quit: Leaving.)
[16:59] * harold (~hamiller@71-94-227-123.dhcp.mdfd.or.charter.com) has joined #ceph
[16:59] * harold (~hamiller@71-94-227-123.dhcp.mdfd.or.charter.com) Quit ()
[17:01] <zenpac> I'm missing some bootstrip magic.
[17:04] * jclm1 (~jclm@172.56.23.106) Quit (Ping timeout: 480 seconds)
[17:05] * zerick (~zerick@irc.quassel.zerick.me) Quit (Quit: http://quassel-irc.org - Chat comfortably. Anywhere.)
[17:07] * TMM (~hp@sams-office-nat.tomtomgroup.com) Quit (Quit: Ex-Chat)
[17:07] * shylesh__ (~shylesh@121.244.87.124) Quit (Remote host closed the connection)
[17:10] * jclm (~jclm@50-206-204-8-static.hfc.comcastbusiness.net) has joined #ceph
[17:10] * SkyEye (~gaveen@123.231.112.221) Quit (Remote host closed the connection)
[17:11] * branto (~branto@ip-213-220-214-203.net.upcbroadband.cz) has joined #ceph
[17:13] * analbeard (~shw@support.memset.com) Quit (Quit: Leaving.)
[17:14] * MentalRay (~MRay@MTRLPQ42-1176054809.sdsl.bell.ca) has joined #ceph
[17:14] * ircolle (~ircolle@50-207-27-138-static.hfc.comcastbusiness.net) has joined #ceph
[17:15] * harold (~hamiller@71-94-227-123.dhcp.mdfd.or.charter.com) has joined #ceph
[17:15] * harold (~hamiller@71-94-227-123.dhcp.mdfd.or.charter.com) Quit ()
[17:20] * kanagaraj (~kanagaraj@nat-pool-bos-t.redhat.com) Quit (Quit: Leaving)
[17:21] * sleinen (~Adium@2001:620:0:82::102) Quit (Read error: Connection reset by peer)
[17:21] * branto (~branto@ip-213-220-214-203.net.upcbroadband.cz) Quit (Ping timeout: 480 seconds)
[17:26] * yanzheng (~zhyan@182.139.20.134) has joined #ceph
[17:26] <zenpac> Is there some way to get an osd in (down, out) mode into (up, in) mode?
[17:27] * rotbeard (~redbeard@185.32.80.238) has joined #ceph
[17:28] * ircolle (~ircolle@50-207-27-138-static.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[17:28] <janos_> zenpac, you cna mark an osd in, but the cluster has to mark it up
[17:29] * ircolle (~ircolle@50-207-27-138-static.hfc.comcastbusiness.net) has joined #ceph
[17:29] <janos_> once you put it in, if everything is good, the cluster will make it in
[17:29] <zenpac> Im getting a superblock error.. Is there a file called "superblock" or is it talking about the partition superblock?
[17:29] <janos_> err, up
[17:29] <zenpac> ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-0: (2) No such file or directory
[17:29] <zenpac> That from ceph-osd.0.log
[17:30] <stj> ... is the disk/osd actually mounted?
[17:30] * ngoswami (~ngoswami@121.244.87.116) has joined #ceph
[17:30] <zenpac> Perhaps it wants a different file structure..
[17:30] * stj scans the scrollback to see if his question is already answered
[17:30] * nhm (~nhm@50-201-39-35-static.hfc.comcastbusiness.net) has joined #ceph
[17:30] * ChanServ sets mode +o nhm
[17:30] <zenpac> yea, it is.
[17:30] * linjan (~linjan@195.110.41.9) has joined #ceph
[17:30] <janos_> and /var/lib/ceph/osd/ceph-0 exists?
[17:30] <zenpac> I'll provide any logs or configs you want..
[17:31] <zenpac> http://paste.debian.net/256546/
[17:32] * Hemanth (~Hemanth@121.244.87.117) Quit (Quit: Leaving)
[17:32] <janos_> not sure. i'm not familiar with that error
[17:32] <stj> there *should* be a superblock file in the ceph-0 directory
[17:32] <stj> and it'll be a ceph thing, rather than your underlying filesystem superblock
[17:35] * topro (~prousa@host-62-245-142-50.customer.m-online.net) Quit (Quit: Konversation terminated!)
[17:35] <zenpac> This is part of the sequence of commans (ansible) addressing osd: http://paste.debian.net/hidden/260b54f7/
[17:36] * kawa2014 (~kawa@89.184.114.246) Quit (Quit: Leaving)
[17:37] * ircolle (~ircolle@50-207-27-138-static.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[17:37] * Nats_ (~natscogs@114.31.195.238) Quit (Read error: Connection reset by peer)
[17:37] * Nats_ (~natscogs@114.31.195.238) has joined #ceph
[17:38] * ircolle (~ircolle@12.191.25.211) has joined #ceph
[17:38] * kefu (~kefu@114.92.100.239) Quit (Ping timeout: 480 seconds)
[17:39] * ska (~skatinolo@cpe-173-174-111-177.austin.res.rr.com) Quit (Ping timeout: 480 seconds)
[17:39] <zenpac> Tried to activate OSD: http://paste.debian.net/256548/
[17:41] * ChrisNBlum (~ChrisNBlu@dhcp-ip-217.dorf.rwth-aachen.de) Quit (Quit: ZNC - http://znc.in)
[17:42] * zerick (~zerick@irc.quassel.zerick.me) has joined #ceph
[17:42] * scuttle|afk is now known as scuttlemonkey
[17:44] * linjan (~linjan@195.110.41.9) Quit (Ping timeout: 480 seconds)
[17:47] * alram (~alram@cpe-172-250-2-46.socal.res.rr.com) has joined #ceph
[17:51] * thomnico (~thomnico@cro38-2-88-180-16-18.fbx.proxad.net) Quit (Quit: Ex-Chat)
[17:53] * ifur (~osm@0001f63e.user.oftc.net) Quit (Quit: leaving)
[17:55] * cholcombe (~chris@c-73-180-29-35.hsd1.or.comcast.net) has joined #ceph
[17:56] * ifur (~osm@0001f63e.user.oftc.net) has joined #ceph
[17:57] <m0zes> zenpac: does /var/lib/ceph/bootstrap-osd/ceph.keyring exist? because that seems to be why ceph-disk activate fails.
[17:57] * vbellur (~vijay@107-1-123-195-ip-static.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[17:58] * ifur (~osm@0001f63e.user.oftc.net) Quit ()
[17:58] <zenpac> No, I create the folders but the keyring is not present.
[17:58] <zenpac> DO I need to manually copy?
[17:59] * dgurtner (~dgurtner@178.197.231.10) Quit (Ping timeout: 480 seconds)
[17:59] <zenpac> I hve copies of /etc/ceph/keyring located in the other /var/lib/ceph/[mon,osd,mds] folders..
[17:59] * zaitcev (~zaitcev@2001:558:6001:10:61d7:f51f:def8:4b0f) has joined #ceph
[17:59] <zenpac> Would that missing file be the same file as /etc/ceph/keyring ?
[18:00] * ifur (~osm@0001f63e.user.oftc.net) has joined #ceph
[18:00] <m0zes> probably need to manually copy it, as it doesn't look like you are using ceph-deploy. I wasn't the one that setup most of my cluster, though.
[18:03] <zenpac> yea, I'm trying to do this manually with a deployment tool..
[18:04] <zenpac> http://paste.debian.net/256585/ (after copying the keyring)
[18:11] * yanzheng (~zhyan@182.139.20.134) Quit (Quit: This computer has gone to sleep)
[18:13] * jtw (~jwilkins@2601:644:4100:bfef:ea2a:eaff:fe08:3f1d) Quit (Ping timeout: 480 seconds)
[18:14] <essjayhch> anyone have any experience of debugging the chef installer for ceph-monitors?
[18:14] * bitserker (~toni@88.87.194.130) Quit (Quit: Leaving.)
[18:14] <essjayhch> because following the install instructions it just breaks.
[18:15] <essjayhch> repeatedly.
[18:22] <zenpac> Do I need /var/lib/ceph/osd/ceph-x for each x, on each host?
[18:22] <zenpac> does ceph-0 need to have a folder for ceph-1 as well?
[18:24] <zenpac> Also, will the automatic tools fail without GPt partitions?
[18:25] * TMM (~hp@178-84-46-106.dynamic.upc.nl) has joined #ceph
[18:25] <rotbeard> zenpac, you just need ../osd/ceph-0 on that host osd 0 is present
[18:25] <rotbeard> zenpac, on host B where OSDs 10-20 exists (for example) you do not need to have ../osd/ceph-0
[18:25] * segutier (~segutier@c-24-6-218-139.hsd1.ca.comcast.net) has joined #ceph
[18:26] * Nacer_ (~Nacer@252-87-190-213.intermediasud.com) has joined #ceph
[18:26] * lx0 (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[18:27] <zenpac> ok.. I thought so, but one of the docs had me guessing.
[18:29] * Nacer_ (~Nacer@252-87-190-213.intermediasud.com) Quit (Remote host closed the connection)
[18:32] * Nacer (~Nacer@252-87-190-213.intermediasud.com) Quit (Ping timeout: 480 seconds)
[18:33] * delattec (~cdelatte@2001:1998:2000:101::16) has joined #ceph
[18:35] * ska (~skatinolo@cpe-173-174-111-177.austin.res.rr.com) has joined #ceph
[18:36] <zenpac> 2015-06-23 11:35:59.290538 7fcc14511700 0 librados: client.bootstrap-osd authentication error (1) Operation not permitted
[18:36] <zenpac> Am I missing some rados auth?
[18:36] * sugoruyo (~georgev@paarthurnax.esc.rl.ac.uk) Quit (Quit: I'm going home!)
[18:38] <zenpac> I'm missing the key but I have keys in most everywhere.
[18:38] * garphy is now known as garphy`aw
[18:39] * cdelatte (~cdelatte@204-235-114.65.twcable.com) Quit (Ping timeout: 480 seconds)
[18:42] * moore (~moore@63-232-3-122.dia.static.qwest.net) has joined #ceph
[18:43] * moore_ (~moore@64.202.160.88) has joined #ceph
[18:44] <zenpac> Do I need a client key in addition to /etc/ceph/keyring.. Perhaps a /etc/ceph/ceph.client.admin.keyring ?
[18:45] * oro (~oro@84-72-180-62.dclient.hispeed.ch) has joined #ceph
[18:45] * rotbeard (~redbeard@185.32.80.238) Quit (Ping timeout: 480 seconds)
[18:48] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[18:50] * Concubidated (~Adium@23.91.33.7) has joined #ceph
[18:50] * moore (~moore@63-232-3-122.dia.static.qwest.net) Quit (Ping timeout: 480 seconds)
[18:51] * sleinen1 (~Adium@2001:620:0:69::102) has joined #ceph
[18:52] <zenpac> Should /var/lib/ceph/bootstrap-osd/{cluster}.keyring be identical to /etc/ceph/keyring?
[18:52] * delatte (~cdelatte@204-235-114.65.twcable.com) has joined #ceph
[18:53] * Concubidated (~Adium@23.91.33.7) Quit ()
[18:55] * dmatson (~david.mat@192.41.52.12) has left #ceph
[18:56] * mgolub (~Mikolaj@91.225.202.121) has joined #ceph
[18:56] * delattec (~cdelatte@2001:1998:2000:101::16) Quit (Read error: Connection reset by peer)
[19:00] * ircolle (~ircolle@12.191.25.211) Quit (Ping timeout: 480 seconds)
[19:01] * arbrandes (~arbrandes@189.78.72.151) Quit (Remote host closed the connection)
[19:03] <zenpac> I don't understand which key is being used whrere...
[19:12] * snakamoto (~Adium@192.16.26.2) has joined #ceph
[19:15] * Nacer (~Nacer@203-206-190-109.dsl.ovh.fr) has joined #ceph
[19:15] * omar_m (~omar_m@12.164.168.117) has joined #ceph
[19:15] * omar_m (~omar_m@12.164.168.117) Quit ()
[19:16] * angdraug (~angdraug@12.164.168.117) has joined #ceph
[19:17] * sleinen (~Adium@2001:620:0:69::103) has joined #ceph
[19:22] * nsoffer (~nsoffer@nat-pool-tlv-t.redhat.com) Quit (Ping timeout: 480 seconds)
[19:22] * sleinen1 (~Adium@2001:620:0:69::102) Quit (Ping timeout: 480 seconds)
[19:24] <zenpac> Do i need special bootstrapping permissions in my keyring?
[19:24] <zenpac> client.bootstrap-osd authentication error (1) Operation not permitted ?
[19:24] * Kupo1 (~tyler.wil@23.111.254.159) has joined #ceph
[19:27] <zenpac> Perhaps related to /usr/bin/python /usr/sbin/ceph-create-keys --cluster ceph -i 0 not finishing
[19:28] * nardial (~ls@dslb-178-011-179-229.178.011.pools.vodafone-ip.de) has joined #ceph
[19:34] * derjohn_mob (~aj@fw.gkh-setu.de) Quit (Ping timeout: 480 seconds)
[19:36] <mongo> I can't find any mention but is there any plan to implement range gets in the radosgw?
[19:37] * t0rn (~ssullivan@2607:fad0:32:a02:56ee:75ff:fe48:3bd3) has joined #ceph
[19:37] * squizzi (~squizzi@2602:306:bc59:85f0:3ea9:f4ff:fe5a:6064) Quit (Quit: to the moon!)
[19:37] * squizzi (~squizzi@2602:306:bc59:85f0:3ea9:f4ff:fe5a:6064) has joined #ceph
[19:37] * t0rn (~ssullivan@2607:fad0:32:a02:56ee:75ff:fe48:3bd3) has left #ceph
[19:40] <mongo> OH, I see it in the API
[19:41] * Be-El (~quassel@fb08-bcf-pc01.computational.bio.uni-giessen.de) Quit (Remote host closed the connection)
[19:42] <SpaceDump> Yay, tomorrow my hardware arrives. Then ?? can finally start playing around with ceph. :D
[19:45] * Hemanth (~Hemanth@117.213.176.211) has joined #ceph
[19:51] * segutier (~segutier@c-24-6-218-139.hsd1.ca.comcast.net) Quit (Quit: segutier)
[19:51] * segutier (~segutier@c-24-6-218-139.hsd1.ca.comcast.net) has joined #ceph
[19:56] * segutier (~segutier@c-24-6-218-139.hsd1.ca.comcast.net) Quit ()
[19:59] * rotbeard (~redbeard@2a02:908:df10:d300:76f0:6dff:fe3b:994d) has joined #ceph
[20:00] * sleinen (~Adium@2001:620:0:69::103) Quit (Ping timeout: 480 seconds)
[20:02] * elder (~elder@104.135.1.105) has joined #ceph
[20:05] * Concubidated (~Adium@192.170.161.35) has joined #ceph
[20:08] * mhack (~mhack@nat-pool-bos-t.redhat.com) Quit (Ping timeout: 480 seconds)
[20:08] * ircolle (~ircolle@64.251.112.55) has joined #ceph
[20:14] * Hemanth (~Hemanth@117.213.176.211) Quit (Ping timeout: 480 seconds)
[20:16] * Hemanth (~Hemanth@117.192.233.71) has joined #ceph
[20:18] * t4nk669 (~oftc-webi@64-7-157-170.border8-dynamic.dsl.sentex.ca) has joined #ceph
[20:19] * mhack (~mhack@nat-pool-bos-u.redhat.com) has joined #ceph
[20:21] <t4nk669> Hi All...lot's to learn about Ceph. I hope you don't mind filling in some gaps I have with the setup. I know it's not recommended to not use SSD drives...but my servers have only SATA bays so I won't be able to add any SSD drives. For each SATA (spinner or OSD) I have in my server, do I put the journal on that same spinner? That way if a SATA drive fails I only lose the one OSD?
[20:23] <burley> can't use a sata ssd in those bays by using a 2.5->3.5" adapter? You could also use a PCIe card for journals if you wanted more perf, but absent that just shove the journal on the same disk as the first partition
[20:24] <monsted> t4nk669: plenty of ways to add SSDs anyway, but yes, put the journal on the same device so you only lose one at a time
[20:28] <t4nk669> Thank you for the replies...hmmm, so I could take one of my SATA bays and add an SSD adapter to it. But then I would need to put all of that servers journallingon the one SSD correct? And if I lose that SSD I lose all my OSD's. Am I understanding this correctly?
[20:29] <rotbeard> t4nk669, yes. but you should put more than _some_ OSD journals to just 1 SSD. even for performance reasons.
[20:30] <rotbeard> _some_ depends on the SSD; for example we use intel dc s3700 with a 1:3 and 1:4 ratio
[20:30] <t4nk669> I'm using infiniband for 40 to connect the servers together. I'm also using a second infiniband port on these Ceph Servers to connect to my Virtual Host Servers. I'm hoping this will give good speed.
[20:30] * ngoswami (~ngoswami@121.244.87.116) Quit (Quit: Leaving)
[20:30] <snakamoto> how many drive bays do your servers have?
[20:30] * arbrandes (~arbrandes@189.78.72.151) has joined #ceph
[20:31] <t4nk669> rotbeard: That's what I thought...if I only use one SSD I'm running the risk of the entire server so I would need more than one SSD on each server and my server only has 8 bays.
[20:32] * ngoswami (~ngoswami@121.244.87.116) has joined #ceph
[20:33] <SpaceDump> 2 ssd, 6 osd... :>
[20:33] <t4nk669> I'm thinking for my setup the best way to go based on the number of bays I have on each server and trying to get the most OSD's I can get out of my cluster (I have 4 servers btw) is to put journalling on each SATA disk. It will not be as fast as using SSD but less risky if I lose a SATA drive..less rebuild time for my data, and I still maximize the number of spinners/OSD's I have for storage capacity. Hoping you can tell if I'
[20:33] <snakamoto> don't forget to keep in mind the performance impact of using your spinning SATA drive for both journal and data
[20:34] <t4nk669> SpaceDump: what about my OS? Do I put that on a SATA disk..or on an SSD? I was thinking of putting my O/S on two SATA drives in Raid1...leaving 6 SATA bays for storage.
[20:34] <t4nk669> Exactly...I do worry about performance. I'm hoping the inifiniband switches I'm using will help with performance greatly even though I'm restricting myself to journals on SATA instead of SSD.
[20:34] <georgem> t4nk669: it depends if you're building your cluster for capacity or performance, in case with low density servers you're already low on capacity
[20:35] <t4nk669> True..I am low on capacity. I'm using 1TB SATA drives in each bay and figure I only need about 4-5 TB of total storage for our small office to run all our VM's from.
[20:36] <t4nk669> To be safe I probably should be looking to use 12 bay servers at least to alleviate these restrictions the 8 bay servers are putting us in now.
[20:36] <t4nk669> Sadly we are repurposing these 4 8 bay servers and don't have the budget to buy more right now.
[20:36] <SpaceDump> t4nk669: True. Didn't think about that part. I'll use a separate sata3 dom-disk for that. (some supermicro thingy) :)
[20:36] * sage (~quassel@2607:f298:6050:709d:58e6:4e74:e4a3:853) Quit (Ping timeout: 480 seconds)
[20:36] <rotbeard> t4nk669, but 40G infiniband wouldn't overcome the slow sata disk performance
[20:37] <rotbeard> SpaceDump, interesting. I just talked about that DOM things yesterday with doppelgrau. are they _good_ enough (maybe in raid1) for the OS?
[20:38] <georgem> t4nk669: you can recover the data from the OSDs backed by a SSD journal, so you can just replace the SSD and ceph will only the update the objects that changed on your osds while they were down, see http://www.sebastien-han.fr/blog/2014/11/27/ceph-recover-osds-after-ssd-journal-failure/
[20:38] * segutier (~segutier@c-24-6-218-139.hsd1.ca.comcast.net) has joined #ceph
[20:38] <SpaceDump> rotbeard: I have no idea yet, I will know for sure in a couple of days.
[20:38] <rotbeard> kk
[20:38] <t4nk669> rotbeard: I was hoping the Infiniband would help with the virtual hosts accessing the data on the Ceph Storage..but you are right...slow SATA drives are slow.
[20:38] <SpaceDump> rotbeard: I'm only going to try it on one of the servers, on the rest I'll use ordinary sata disk, just because I'm not sure about the reliability yet. :/
[20:39] <rotbeard> SpaceDump, same here. but I have some of the shiny X10 boards from supermicro. so I have a lot of onboard sata ports. maybe I could use 6 DOMs. raid1 + spare :D
[20:39] <snakamoto> with only 6 bays available, a 1:5 ratio for SSD journal to OSD is not bad
[20:40] <SpaceDump> rotbeard: Same here. :D X10SRW-F
[20:40] <SpaceDump> rotbeard: And a chassi with 10 slots for disks. :D
[20:40] <rotbeard> snakamoto, ack. but someone needs to be sure to use very good and stable enterprise ssds
[20:40] * mhack (~mhack@nat-pool-bos-u.redhat.com) Quit (Ping timeout: 480 seconds)
[20:40] <rotbeard> SpaceDump, we have those 16 + 2 (back) bay chassis
[20:40] <t4nk669> georgem: Ok..that's good to know. So for my small 8 bay servers would the following be best for us then...still use 2 bays for the O/S (in Raid1). use 4 bays of 1TB drives for OSD/spinners and use 2 bays with an SSD adpater for the journals. Each SSD drive handles the journals of 2 SATA OSD's?
[20:41] <SpaceDump> rotbeard: Oh, sweet. I just tried to keep the cost down this time. Think this should be enough. I really hope, can't afford anything else at the moment..
[20:41] <t4nk669> Or do I only need 1 SSD drive for 5 OSD?
[20:41] * ira (~ira@c-71-233-225-22.hsd1.ma.comcast.net) Quit (Remote host closed the connection)
[20:43] * ChrisHolcombe (~chris@c-73-180-29-35.hsd1.or.comcast.net) has joined #ceph
[20:43] * cholcombe (~chris@c-73-180-29-35.hsd1.or.comcast.net) Quit (Read error: Connection reset by peer)
[20:43] * ira (~ira@c-71-233-225-22.hsd1.ma.comcast.net) has joined #ceph
[20:43] <doppelgrau> t4nk669: depends on the SSD (IO/s with sync IO, and how durable the SSD is).
[20:44] * ngoswami (~ngoswami@121.244.87.116) Quit (Quit: Leaving)
[20:44] * dneary (~dneary@64.251.112.55) has joined #ceph
[20:45] <doppelgrau> t4nk669: yesterday someone was in the channel with an all-SSD setup with terrible performance since the used model did fall down to a few IO/s with small synchronous IO
[20:45] * segutier (~segutier@c-24-6-218-139.hsd1.ca.comcast.net) Quit (Quit: segutier)
[20:47] <snakamoto> t4nk669: You'll have to weigh some pros and cons as well. Some basic ones: with 1:5 you get more capacity, but the SSD might be a performance bottleneck. Also, with 1:5, if an SSD fails, you're going to have more data that needs to be redistributed or calculated than with a 1:2 ratio.
[20:48] <doppelgrau> t4nk669: If you know the IO per second you need to handle, you can calculate if you need SSD or not
[20:49] <doppelgrau> t4nk669: Really depends on the workload what makes sense
[20:49] <georgem> t4nk669: Yes, your setup seems sane; 2 small sata drives in a raid 1 for OS, 1 or 2 SSDs and the rest SATA OSDs; with 4 servers you would have 16 TB raw in the cluster with more than 5 TB in a replica 3 pool; if your budget and performance looks good you can do this setup
[20:50] * mhack (~mhack@nat-pool-bos-t.redhat.com) has joined #ceph
[20:51] <t4nk669> Thank you all for your comments. Very good information here. Yes, I can see how the performance I 'Might' need depends on the IO per cluster or what my virtual hosts need from the Ceph Storage. But to be safe, and based on the storage we need which isn't huge, I do think having 2 SSD's and 4 OSD's per server would be best for us.
[20:51] <zenpac> Is /etc/ceph/rbdmap required ?
[20:51] * jtw (~jwilkins@2601:644:4100:bfef:ea2a:eaff:fe08:3f1d) has joined #ceph
[20:51] <t4nk669> georgem: Thank you for your comments. Much appreciated. Glad to see I have a better plan now on how to proceed.
[20:51] <georgem> t4nk669:two decent SSDs can write about 800 MB/s, the OSDs behind them can write about 4-500 MB/s in total and the network can bring in 5000 MB/s, so your bottleneck will be the SATA OSDs
[20:52] * nhm (~nhm@50-201-39-35-static.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[20:52] <t4nk669> And for performance...can anyone suggest the best SSD for price they would recommend for a small office like us? :)
[20:52] <georgem> t4nk669: Intel S3700 is what everybody seems to be using
[20:53] * oro (~oro@84-72-180-62.dclient.hispeed.ch) Quit (Ping timeout: 480 seconds)
[20:53] <snakamoto> zenpac: did you get the issue with osd bootstrap auth worked out?
[20:53] <georgem> t4nk669: you can see how much it would cost you with two SSDs per server vs. one and go from there
[20:55] * Hemanth (~Hemanth@117.192.233.71) Quit (Quit: Leaving)
[20:55] * nsoffer (~nsoffer@bzq-79-180-80-9.red.bezeqint.net) has joined #ceph
[20:58] * segutier (~segutier@c-24-6-218-139.hsd1.ca.comcast.net) has joined #ceph
[20:59] * ChrisNBlum (~ChrisNBlu@dhcp-ip-217.dorf.rwth-aachen.de) has joined #ceph
[20:59] * sage (~quassel@2607:f298:6050:709d:f030:67aa:1c6f:a15) has joined #ceph
[20:59] * ChanServ sets mode +o sage
[20:59] <zenpac> snakamoto: no.. I'm still working on it..
[20:59] <zenpac> I have it narrowed down to authentication issue.
[20:59] <zenpac> http://paste.debian.net/257115/
[21:00] <t4nk669> Thank you all for your suggestions today. Very much appreciated!
[21:00] <snakamoto> I'm not entirely familiar but hopefully I can help somehow
[21:00] <snakamoto> zenpac: are you trying to initialize the very first OSD on a new cluster?
[21:01] <zenpac> Yes
[21:01] <t4nk669> Sorry, one more question...how big of an SSD do I need if I use two per server in my setup with 1 TB drives? Or how big does a journal get per OSD?
[21:02] <zenpac> /var/lib/ceph/bootstrap-osd/ceph.keyring seems to be missing, but that is what ceph-activate is trying to do.
[21:02] <snakamoto> t4nk669: that's a tunable setting. I believe default is 5GB per OSD
[21:02] <nils_> 600 active+undersized+degraded
[21:03] <nils_> gotta love that upgrade adrenaline rush
[21:03] <snakamoto> zenpac: On your mon server, that ceph.keyring file should exist in the same path. (/var/lib/ceph/boostrap-osd)
[21:03] <snakamoto> You should be able to just copy it from there
[21:03] <t4nk669> Do these Intel S3700 come in small sizes? Like 100 GB? Looks like all I need are two small SSD drives then?
[21:04] <nils_> 80G I think but those are slow.
[21:05] <rotbeard> t4nk669, we use 200G s3700 to get the needed performance. + my journals are ~20G big (pretty optimistic) so I have 3 or 4 20G partitions + the remaining space for that wear leveling magic
[21:06] * ircolle (~ircolle@64.251.112.55) Quit (Ping timeout: 480 seconds)
[21:06] <zenpac> snakamoto: its not mounted yet.. I guess I need to mount it.
[21:07] <zenpac> snakamoto: I'm not seeing that file. How is it made?
[21:08] <zenpac> snakamoto: can I copy the /etc/ceph/keyring into that file?
[21:08] <zenpac> I'm very confused by the intense keyring concept.
[21:10] <snakamoto> no, that's not the correct keyring
[21:10] <snakamoto> do you have a monitor server running yet?
[21:12] <zenpac> snakamoto: yes,.. But my nodes all are same, each has (mon, mds, osd) on them.
[21:13] <zenpac> But ceph-mon is running.
[21:13] <zenpac> I see mon ports listening ok.
[21:13] <t4nk669> rotbeard and nils: Thanks both for your comments. yes I do see these Intel S3700 come in 200 GB sizes. More than enough for us and they have the performance.
[21:13] * madkiss1 (~madkiss@2001:6f8:12c3:f00f:9837:7492:a792:20f2) Quit (Ping timeout: 480 seconds)
[21:15] <snakamoto> zenpac: run 'ceph auth list' towards the bottom you should see a key called client.bootstrap-osd
[21:15] * TMM (~hp@178-84-46-106.dynamic.upc.nl) Quit (Quit: Ex-Chat)
[21:16] <todin> 7win 1
[21:17] * vbellur (~vijay@50-206-204-8-static.hfc.comcastbusiness.net) has joined #ceph
[21:17] <zenpac> snakamoto: I don not..
[21:17] <zenpac> Its not there. I didn't probably make it..
[21:17] <zenpac> I'll show you what it lists...
[21:18] <zenpac> snakamoto: http://paste.debian.net/257121/
[21:19] * analbeard (~shw@5.80.205.222) has joined #ceph
[21:21] <zenpac> snakamoto: was I supposed to create such a key ?
[21:22] * dneary (~dneary@64.251.112.55) Quit (Ping timeout: 480 seconds)
[21:24] <zenpac> snakamoto: ceph auth get-or-create-key client.bootstrap-osd mon "allow profile bootstrap-osd" (seems to add the right key in /etc/ceph/keyring)...
[21:27] * nhm (~nhm@172.56.4.7) has joined #ceph
[21:27] * ChanServ sets mode +o nhm
[21:31] * vbellur (~vijay@50-206-204-8-static.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[21:31] * nardial (~ls@dslb-178-011-179-229.178.011.pools.vodafone-ip.de) Quit (Quit: Leaving)
[21:33] * ndru (~jawsome@ec2-54-164-112-154.compute-1.amazonaws.com) has joined #ceph
[21:34] * snakamoto (~Adium@192.16.26.2) Quit (Quit: Leaving.)
[21:37] * nils_ (~nils@doomstreet.collins.kg) Quit (Quit: This computer has gone to sleep)
[21:38] * i_m (~ivan.miro@deibp9eh1--blueice4n1.emea.ibm.com) Quit (Ping timeout: 480 seconds)
[21:40] * snakamoto (~Adium@192.16.26.2) has joined #ceph
[21:41] * sleinen (~Adium@84-72-160-233.dclient.hispeed.ch) has joined #ceph
[21:41] <snakamoto> zenpac: I'm not sure how you installed. I've done the manual install (not using ceph-deploy) a handful of times, and it's always created the key for me.
[21:42] * mfa298 (~mfa298@gateway.yapd.net) has joined #ceph
[21:42] <zenpac> snakamoto: what directions did you use?
[21:42] <snakamoto> that command appears correct. My key's caps is "caps: [mon] allow profile bootstrap-osd"
[21:43] <zenpac> snakamoto: is there a script or a document you use?
[21:43] * sleinen1 (~Adium@2001:620:0:82::105) has joined #ceph
[21:43] <snakamoto> I've basically followed this: http://ceph.com/docs/v0.80.5/install/manual-deployment/
[21:43] <ndru> Hello all, Working on implementing ceph into my existing openstack infrastructure which is currently using cinder-volumes. I understand ceph will need to overwrite my raid array. I'd like to bring ceph up on one cinder node so I can copy my existing data into it before adding the current cinder node to the ceph cluster. Can I temporarily have replicas set to 1? Or is there a recommended way to handle that procedure?
[21:44] * dmick (~dmick@206.169.83.146) has left #ceph
[21:44] <zenpac> snakamoto: do you use a deployment tool or is just manual?
[21:45] <snakamoto> I've done mostly manual. I did write a very very simple Puppet module to do a basic install.
[21:47] <zenpac> What I wantedt to do was a simple 3-node deployment where each node had all (mon, mds, ods) servers on it , strictly for testing..
[21:47] * TMM (~hp@178-84-46-106.dynamic.upc.nl) has joined #ceph
[21:49] * sleinen (~Adium@84-72-160-233.dclient.hispeed.ch) Quit (Ping timeout: 480 seconds)
[21:53] * mgolub (~Mikolaj@91.225.202.121) Quit (Quit: away)
[21:53] * fdmanana__ (~fdmanana@bl13-155-180.dsl.telepac.pt) Quit (Ping timeout: 480 seconds)
[21:59] * dopesong (~dopesong@lb1.mailer.data.lt) has joined #ceph
[22:05] * delatte (~cdelatte@204-235-114.65.twcable.com) Quit (Quit: This computer has gone to sleep)
[22:05] * xdeller (~xdeller@h195-91-128-218.ln.rinet.ru) Quit (Read error: Connection reset by peer)
[22:05] * nhm (~nhm@172.56.4.7) Quit (Ping timeout: 480 seconds)
[22:14] * Nacer (~Nacer@203-206-190-109.dsl.ovh.fr) Quit (Read error: Connection reset by peer)
[22:15] * Nacer (~Nacer@203-206-190-109.dsl.ovh.fr) has joined #ceph
[22:16] * squizzi (~squizzi@2602:306:bc59:85f0:3ea9:f4ff:fe5a:6064) Quit (Quit: to the moon!)
[22:16] * squizzi (~squizzi@2602:306:bc59:85f0:3ea9:f4ff:fe5a:6064) has joined #ceph
[22:19] * ganders (~root@190.2.42.21) Quit (Quit: WeeChat 0.4.2)
[22:20] * TMM (~hp@178-84-46-106.dynamic.upc.nl) Quit (Quit: Ex-Chat)
[22:22] * analbeard (~shw@5.80.205.222) Quit (Ping timeout: 480 seconds)
[22:23] * TMM (~hp@178-84-46-106.dynamic.upc.nl) has joined #ceph
[22:23] * dopesong (~dopesong@lb1.mailer.data.lt) Quit (Remote host closed the connection)
[22:29] * snakamoto (~Adium@192.16.26.2) Quit (Quit: Leaving.)
[22:35] * dopesong (~dopesong@lb1.mailer.data.lt) has joined #ceph
[22:37] * ChrisHolcombe (~chris@c-73-180-29-35.hsd1.or.comcast.net) Quit (Remote host closed the connection)
[22:41] * Sysadmin88 (~IceChat77@2.124.164.69) has joined #ceph
[22:42] * cholcombe (~chris@c-73-180-29-35.hsd1.or.comcast.net) has joined #ceph
[22:43] * jdillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) Quit (Quit: jdillaman)
[22:44] * wicope (~wicope@0001fd8a.user.oftc.net) Quit (Remote host closed the connection)
[22:45] * georgem (~Adium@fwnat.oicr.on.ca) Quit (Quit: Leaving.)
[22:46] * scuttlemonkey is now known as scuttle|afk
[22:46] * snakamoto (~Adium@192.16.26.2) has joined #ceph
[22:49] * jdillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) has joined #ceph
[22:51] * shohn (~shohn@dslb-088-075-235-250.088.075.pools.vodafone-ip.de) Quit (Quit: Leaving.)
[22:52] * jdillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) Quit ()
[22:52] * sleinen1 (~Adium@2001:620:0:82::105) Quit (Quit: Leaving.)
[22:54] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[22:58] * dopesong (~dopesong@lb1.mailer.data.lt) Quit (Remote host closed the connection)
[23:01] * badone (~brad@CPE-121-215-241-179.static.qld.bigpond.net.au) has joined #ceph
[23:03] * TMM (~hp@178-84-46-106.dynamic.upc.nl) Quit (Quit: Ex-Chat)
[23:04] * MentalRay (~MRay@MTRLPQ42-1176054809.sdsl.bell.ca) Quit (Quit: This computer has gone to sleep)
[23:05] * tupper (~tcole@2001:420:2280:1272:8900:f9b8:3b49:567e) Quit (Ping timeout: 480 seconds)
[23:17] * yguang11 (~yguang11@nat-dip30-wl-d.cfw-a-gci.corp.yahoo.com) has joined #ceph
[23:17] * daviddcc (~dcasier@84.197.151.77.rev.sfr.net) Quit (Ping timeout: 480 seconds)
[23:25] * snakamoto (~Adium@192.16.26.2) Quit (Quit: Leaving.)
[23:26] * oro (~oro@84-72-180-62.dclient.hispeed.ch) has joined #ceph
[23:30] * bobrik (~bobrik@83.243.64.45) Quit (Quit: (null))
[23:44] * MentalRay (~MRay@MTRLPQ42-1176054809.sdsl.bell.ca) has joined #ceph
[23:49] * dyasny (~dyasny@104.158.35.152) Quit (Ping timeout: 480 seconds)
[23:51] * qhartman (~qhartman@den.direwolfdigital.com) has joined #ceph
[23:53] <qhartman> I'm looking at reducing my OSD IO in my ceph cluster if possible, and I'm looking at increasing filestore_max_sync_interval to see if that will allow for less frequent, but larger (sequential?) IO on the OSD disk
[23:54] <qhartman> but I would like to see how full my journals are getting before messing with that setting to make sure I have the headroom for the additional space that will presuambly be used by the journals
[23:54] <qhartman> How can I find that information? Since ceph just writes directly tot he raw partition I don't know of a way to discover that at the system level, and I have found no ceph commands that expose that.
[23:55] <qhartman> Am I missing something, or is it truly not really possible?
[23:56] * fdmanana__ (~fdmanana@bl13-155-180.dsl.telepac.pt) has joined #ceph
[23:59] <qhartman> I've also noticed that the filestore_journal_writeahead setting is is set to "false", though that is supposed to default to true for XFS filestores, which I am using.
[23:59] <qhartman> Is that setting just appearing to be wrong and that it what ceph would be using internally since I have XFS filestores?

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.