#ceph IRC Log

Index

IRC Log for 2014-07-03

Timestamps are in GMT/BST.

[0:00] * rweeks (~rweeks@c-24-6-118-113.hsd1.ca.comcast.net) Quit (Quit: Leaving)
[0:01] * rendar (~I@95.234.176.146) Quit ()
[0:03] * ircolle is now known as ircolle-away
[0:10] * aldavud (~aldavud@217-162-119-191.dynamic.hispeed.ch) has joined #ceph
[0:23] * hufman (~hufman@cpe-184-58-235-28.wi.res.rr.com) Quit (Quit: leaving)
[0:24] * joef1 (~Adium@c-67-188-220-98.hsd1.ca.comcast.net) has joined #ceph
[0:25] * joef (~Adium@2620:79:0:131:b903:7c55:6c42:32aa) Quit (Remote host closed the connection)
[0:26] * fsimonce (~simon@host27-60-dynamic.26-79-r.retail.telecomitalia.it) Quit (Quit: Coyote finally caught me)
[0:34] * koleosfuscus (~koleosfus@adsl-84-226-68-69.adslplus.ch) Quit (Quit: koleosfuscus)
[0:39] * Midnightmyth (~quassel@93-167-84-102-static.dk.customer.tdc.net) Quit (Ping timeout: 480 seconds)
[0:48] * primechu_ (~primechuc@host-95-2-129.infobunker.com) Quit (Remote host closed the connection)
[0:49] * csaesumu_ (~quassel@102.92.14.37.dynamic.jazztel.es) Quit (Remote host closed the connection)
[0:57] <MACscr> should a cache pool be replicated?
[1:00] <Gugge-47527> yes
[1:00] <Gugge-47527> if you like your data
[1:01] <Gugge-47527> it is not just a read cache
[1:01] * zack_dolby (~textual@pdf8519e7.tokynt01.ap.so-net.ne.jp) Quit (Quit: My MacBook has gone to sleep. ZZZzzz???)
[1:01] * sigsegv (~sigsegv@188.25.120.56) Quit (Quit: sigsegv)
[1:05] <iggy> what if you're not particularly fond of your data because one time when it was drunk it said something about wanting to bang your sister?
[1:05] <MACscr> great point
[1:09] <MACscr> whats the proper way to apply to conf settings and crushmap settings? I know the crushmap is imported and you can push out the conf, but are they applied right away or does the ceph service need restarted?
[1:10] * hasues (~hasues@kwfw01.scrippsnetworksinteractive.com) Quit (Quit: Leaving.)
[1:12] <iggy> ceph wouldn't be very useful if you have to shut it down everytime you changed something with the config
[1:12] <Gugge-47527> the conf file is only read when the daemons start
[1:13] <iggy> while that is correct... https://ceph.com/docs/master/rados/configuration/ceph-conf/#runtime-changes
[1:15] * Nats__ (~Nats@2001:8000:200c:0:ed20:896c:d0a4:9f54) has joined #ceph
[1:16] * yguang11 (~yguang11@vpn-nat.corp.tw1.yahoo.com) Quit ()
[1:18] * AfC (~andrew@nat-gw2.syd4.anchor.net.au) has joined #ceph
[1:19] * john (~john@2600:1010:b013:5417:2ce2:b5a7:1a82:7050) Quit (Quit: Leaving)
[1:20] * sz0_ (~sz0@46.197.48.116) has joined #ceph
[1:22] * Nats_ (~Nats@2001:8000:200c:0:ccb0:7f7e:aca8:ad13) Quit (Ping timeout: 480 seconds)
[1:24] * gillesMo (~gillesMo@00012912.user.oftc.net) Quit (Quit: Konversation terminated!)
[1:25] * LeaChim (~LeaChim@host109-146-189-84.range109-146.btcentralplus.com) Quit (Read error: Operation timed out)
[1:28] * aldavud (~aldavud@217-162-119-191.dynamic.hispeed.ch) Quit (Ping timeout: 480 seconds)
[1:37] * kevinc (~kevinc__@client65-78.sdsc.edu) Quit (Quit: This computer has gone to sleep)
[1:39] * kevinc (~kevinc__@client65-78.sdsc.edu) has joined #ceph
[1:42] * joef1 (~Adium@c-67-188-220-98.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[1:43] * joef (~Adium@c-67-188-220-98.hsd1.ca.comcast.net) has joined #ceph
[1:43] * joef (~Adium@c-67-188-220-98.hsd1.ca.comcast.net) Quit ()
[1:46] * sputnik13 (~sputnik13@207.8.121.241) Quit (Quit: My MacBook has gone to sleep. ZZZzzz???)
[1:53] * jtaguinerd (~jtaguiner@125.212.121.243) has joined #ceph
[1:54] * kevinc (~kevinc__@client65-78.sdsc.edu) Quit (Quit: This computer has gone to sleep)
[1:56] * kevinc (~kevinc__@client65-78.sdsc.edu) has joined #ceph
[2:00] <MACscr> iggy: thats just individual settings though isnt it?
[2:00] <MACscr> not a whole conf worth of changes
[2:01] <MACscr> more for just temporary changes
[2:02] <MACscr> though i guess most changes are in the crushmap that happen immediately?
[2:02] <MACscr> when its imported?
[2:03] * zack_dolby (~textual@e0109-49-132-43-197.uqwimax.jp) has joined #ceph
[2:03] * sjusthm (~sam@66-214-251-229.dhcp.gldl.ca.charter.com) Quit (Quit: Leaving.)
[2:09] * kevinc (~kevinc__@client65-78.sdsc.edu) Quit (Quit: This computer has gone to sleep)
[2:17] * aldavud (~aldavud@217-162-119-191.dynamic.hispeed.ch) has joined #ceph
[2:18] * huangjun (~kvirc@111.174.239.37) has joined #ceph
[2:26] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit (Quit: Leaving.)
[2:28] * xarses (~andreww@12.164.168.117) Quit (Read error: Operation timed out)
[2:31] <iggy> if you are having to change out large parts of your ceph.conf frequently, you're doing it wrong
[2:32] * aldavud (~aldavud@217-162-119-191.dynamic.hispeed.ch) Quit (Read error: Operation timed out)
[2:45] * bitserker (~toni@63.pool85-52-240.static.orange.es) Quit (Read error: Operation timed out)
[2:46] * bitserker (~toni@63.pool85-52-240.static.orange.es) has joined #ceph
[2:52] * pressureman (~pressurem@62.217.45.26) Quit (Ping timeout: 480 seconds)
[2:52] * stephan (~stephan@62.217.45.26) Quit (Ping timeout: 480 seconds)
[2:57] * pressureman (~pressurem@62.217.45.26) has joined #ceph
[2:57] * stephan (~stephan@62.217.45.26) has joined #ceph
[3:07] * cookednoodles (~eoin@eoin.clanslots.com) Quit (Remote host closed the connection)
[3:09] * narb (~Jeff@38.99.52.10) Quit (Quit: narb)
[3:18] * yguang11 (~yguang11@2406:2000:ef96:e:14dd:678c:435:6dfc) has joined #ceph
[3:20] * xarses (~andreww@c-24-23-183-44.hsd1.ca.comcast.net) has joined #ceph
[3:22] * zerick (~eocrospom@190.187.21.53) Quit (Ping timeout: 480 seconds)
[3:27] * lucas1 (~Thunderbi@218.76.25.66) has joined #ceph
[3:34] * zhaochao (~zhaochao@124.207.139.17) has joined #ceph
[3:34] * Guest541 (~rweeks@c-24-6-118-113.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[3:36] <Jakey> hi i am having problems this is my error http://pastebin.com/tbUqK4d2
[3:36] <Jakey> this is my config https://www.irccloud.com/pastebin/nlspYQ4n
[3:53] * jcsp1 (~Adium@82-71-55-202.dsl.in-addr.zen.co.uk) has joined #ceph
[3:57] * jcsp (~Adium@0001bf3a.user.oftc.net) Quit (Ping timeout: 480 seconds)
[4:02] * lucas1 (~Thunderbi@218.76.25.66) Quit (Ping timeout: 480 seconds)
[4:05] * Pedras (~Adium@216.207.42.140) Quit (Read error: Connection reset by peer)
[4:35] * wschulze (~wschulze@cpe-69-206-251-158.nyc.res.rr.com) has joined #ceph
[4:36] * diegows (~diegows@190.190.5.238) Quit (Read error: Operation timed out)
[4:39] * bkopilov (~bkopilov@213.57.16.107) Quit (Ping timeout: 480 seconds)
[4:43] * lucas1 (~Thunderbi@222.247.57.50) has joined #ceph
[4:45] * angdraug (~angdraug@12.164.168.117) Quit (Quit: Leaving)
[4:48] * lucas1 (~Thunderbi@222.247.57.50) Quit ()
[4:48] * Cube (~Cube@66-87-65-137.pools.spcsdns.net) Quit (Read error: No route to host)
[4:50] * wschulze (~wschulze@cpe-69-206-251-158.nyc.res.rr.com) Quit (Quit: Leaving.)
[4:51] <MACscr> Jakey: only one monitor?
[4:51] <MACscr> how mayn osd hosts?
[4:51] <MACscr> many
[4:52] * Cube (~Cube@66-87-66-1.pools.spcsdns.net) has joined #ceph
[4:52] <Jakey> i doing the preflight MACscr
[4:52] <Jakey> please help
[4:53] <Jakey> um.. yes i want to setup one monitor first with 2 osd
[4:53] <MACscr> Jakey: im only a day into using it, so im no expert
[4:53] <Jakey> MACscr: did you manage to get pass that error?
[4:53] <Jakey> MACscr: i have no idea what the fuck is going on
[4:53] <MACscr> not a good idea. i didnt get that error. im doing 3 nodes with 3 monitors
[4:54] <MACscr> so the monitors and osd's are on the same server
[4:54] <MACscr> a single monitor is not a good idea and 2 node setups are not recommended
[4:55] <Jakey> i'm following this http://ceph.com/docs/master/start/quick-ceph-deploy/
[4:55] <Jakey> MACscr: are you using centos 5.4
[4:55] <Jakey> 6.4
[4:55] <MACscr> nope, debian 7.5
[4:56] <Jakey> okay maybe thats why
[4:56] <MACscr> also, 6.5 is the newest isnt it?
[4:56] <MACscr> why would you be on 6.4
[4:56] <Jakey> yeah so?
[4:57] <MACscr> also, what version of ceph? i do know that ceph-deploy is now on version 1.5.7 and you are on 1.5.5
[4:57] <MACscr> though 1.5.5 wasnt to bad
[4:58] <MACscr> also, that quickstart guide does say 3 nodes
[4:59] <MACscr> and mon on each one
[5:01] <JCL> Jakey: Fix your name resolution so that your host always resolve to the same name: ibm
[5:01] <JCL> In one case it resolves to ibm
[5:01] <JCL> In another case to ibm.node2.local
[5:02] <JCL> Everything must match and be always the same value
[5:02] * Vacum_ (~vovo@88.130.223.54) has joined #ceph
[5:02] <Jakey> JCL: this is my /etc/hosts on every server
[5:02] <Jakey> 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
[5:02] <Jakey> 192.168.50.6 ibm.node2.local
[5:02] <Jakey> 192.168.50.9 ibm.node1.local
[5:02] <Jakey> 192.168.50.23 node6.local
[5:02] <Jakey> 192.168.50.17 node2.local
[5:03] <MACscr> you need the shortname on there too
[5:03] <JCL> Yes
[5:03] <Jakey> huh?
[5:04] <JCL> node1, node2 not funky semi fqdn
[5:04] <JCL> And make sure its matches the output of hostname -s command
[5:04] <Jakey> JCL: well those ibm node names resolve to just ibm
[5:04] <Jakey> the same
[5:05] <JCL> From your pastebin : processing monitor mon.ibm.node2.local
[5:05] <JCL> So no they do not
[5:05] <JCL> Make sure it's all consistent acroos your nodes. Without this. no easy way out.
[5:05] <Jakey> jcl yes its all consistent
[5:06] <Jakey> JCL: it resolves to mone.node6
[5:06] <Jakey> than it trys to connect to mon.node6.local again
[5:06] <Jakey> i don't get it
[5:06] <Jakey> from the logs look at it
[5:06] <Jakey> it got the mon.node6 running
[5:07] <Jakey> and then it trys to connect to mon.node6.local which theres no socket for it
[5:07] * rturk-away (~rturk@ds2390.dreamservers.com) Quit (Read error: Operation timed out)
[5:07] <JCL> So may be check the difference between the one that kicks and the others
[5:07] * rturk-away (~rturk@ds2390.dreamservers.com) has joined #ceph
[5:07] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[5:07] <Jakey> JCL: i think i should try to shorten the hostnames with no periods?
[5:07] <Jakey> what if that doesn't work?
[5:07] <Jakey> but i will check it out
[5:08] <JCL> Removing the dots surely is a good idea. ;-)
[5:08] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit (Read error: Connection reset by peer)
[5:09] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[5:09] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit (Remote host closed the connection)
[5:09] * Vacum (~vovo@88.130.222.57) Quit (Ping timeout: 480 seconds)
[5:10] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[5:10] <Jakey> okay i will try everything from the start again and see if it fix it thanks alot JCL
[5:10] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit ()
[5:11] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[5:13] <MACscr> Jakey: if you are just testing internally, so something more like mon1.foobar.local
[5:13] <MACscr> then you can just have mon1 as your shortname
[5:22] * rongze (~rongze@114.54.30.94) has joined #ceph
[5:28] <MACscr> JCL: i havent imported it yet, but this is my custom crushmap that i have created so far in order 2 seperate the OSD items into two pools. Does this look correct to you? http://pastie.org/pastes/9348826/text?key=xgmb0zcz6xnayjnuqvh5xq
[5:28] * lucas1 (~Thunderbi@222.240.148.130) has joined #ceph
[5:29] <dmick> MACscr: you probably want those chooseleaf stmts to be "type osd"
[5:29] * lucas1 (~Thunderbi@222.240.148.130) Quit ()
[5:29] <MACscr> ha, thanks. I was literally just reading about that
[5:29] <dmick> and you want the roots to refer to the hosts, not the osds
[5:30] <dmick> (it's a tree, or in your case two trees)
[5:30] <dmick> although...actually
[5:30] <dmick> hm
[5:30] <dmick> no, forget I said that second bit
[5:31] * davidzlap (~Adium@ip68-4-173-198.oc.oc.cox.net) Quit (Quit: Leaving.)
[5:31] <MACscr> i was wondering about that too, but the osd's are on the same hosts
[5:31] <MACscr> so i hoping that can still be split up
[5:31] <MACscr> each host has 3 osd's. Wanting two of those for the ssd pool and 1 for the platter pool
[5:31] <dmick> yeah, I don't know if you can do that
[5:32] <MACscr> man, if thats the case, i could be screwed and wouldnt really even be able to do a cache pool then
[5:32] <dmick> I mean effectively the hosts don't even enter into your map the way it's written
[5:32] <MACscr> true
[5:32] <dmick> so it might work, but wouldn't be able to deal with the host failure domain
[5:33] <dmick> I would use crushtool's 'test" functionality to do some simulation
[5:33] <MACscr> well wouldnt it just fall under the osd failure for each one then?
[5:33] <dmick> the thing is, if you don't tell crush these things are on different hosts
[5:34] <dmick> it's free to choose all replicas on one host
[5:34] * wschulze (~wschulze@cpe-69-206-251-158.nyc.res.rr.com) has joined #ceph
[5:34] <dmick> which means that host failing means data loss
[5:34] <MACscr> ah, good point. hmm
[5:34] <MACscr> i do have them listed on different hosts, but obviously no rules for that
[5:34] * KaZeR (~kazer@64.201.252.132) Quit (Read error: Operation timed out)
[5:34] <MACscr> hmm
[5:35] <MACscr> is it possible ot manually map the osd replication from one osd to another?
[5:35] <dmick> not sure what you mean
[5:36] <dmick> object -> crush -> set of devices to store replicas on, pseudo-random
[5:40] <MACscr> so i applied it and here is what my osd tree looks like now: http://pastie.org/private/4alji2qiog7bitvcjrnhg
[5:43] <MACscr> now if im just using ceph for rbd, do i really need pools data and metadata? Seems like data is being used by cephfs, which i really dont think i need
[5:44] <dmick> no, but they're also not hurting anything
[5:48] <MACscr> hmm, so why does it say that pools data and metadata are being used by cephfs.
[5:48] <MACscr> i havent configured anything for cephfs afaik
[5:48] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[5:48] <dmick> they're there for cephfs by dfeault
[5:49] <lurbs> Instead of splitting OSDs by type into different root, would it be easier if there was functionality to let Ceph maintain the tree, and add some sort of type tags to the OSDs (SSD/spinner) etc?
[5:49] <dmick> there have been a few issues in recent builds with removing them
[5:49] <lurbs> s/add/adding/
[5:49] <kraken> lurbs meant to say: Instead of splitting OSDs by type into different root, would it be easier if there was functionality to let Ceph maintain the tree, and adding some sort of type tags to the OSDs (SSD/spinner) etc?
[5:51] * davidzlap (~Adium@ip68-4-173-198.oc.oc.cox.net) has joined #ceph
[5:58] <JCL> MACscr: That map will be useless is production as you will not be protected against a host failure. The root inlude only OSDs in both case and your rules will have to pick up a choose type host to get to work, making it possible to distribute the n copies across OSDs that live in the same box
[5:58] <MACscr> so basically there can only be a single pool per host?
[5:59] <JCL> This mechanism can only be achieve by introducing "phantom" host for each host: hostrealname containing the platetr OSDs and hostssd containing the SSD OSDs.
[5:59] <JCL> s/achieve/achieved/
[5:59] <kraken> JCL meant to say: This mechanism can only be achieved by introducing "phantom" host for each host: hostrealname containing the platetr OSDs and hostssd containing the SSD OSDs.
[5:59] <MACscr> ah, great idea
[5:59] <MACscr> thanks
[5:59] <dmick> JCL: ah, that makes sense
[6:00] <JCL> You ought to start walking before you want to run my gran'pa use to say ;-)
[6:00] <MACscr> but i was already walking???.
[6:00] <dmick> everything's clear from the right vantage point
[6:01] <JCL> Dunno :-)
[6:01] * bkopilov (~bkopilov@nat-pool-tlv-t.redhat.com) has joined #ceph
[6:02] <lurbs> JCL: Yeah, that's how we're splitting them at the moment. Seems to work, even if there is a bit of maintenance overhead.
[6:04] * Guest243 (~coyo@209.148.95.237) Quit (Remote host closed the connection)
[6:09] * Coyo (~coyo@209.148.95.237) has joined #ceph
[6:09] * Coyo is now known as Guest625
[6:09] * wschulze (~wschulze@cpe-69-206-251-158.nyc.res.rr.com) Quit (Quit: Leaving.)
[6:11] * Cube (~Cube@66-87-66-1.pools.spcsdns.net) Quit (Ping timeout: 480 seconds)
[6:13] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit (Quit: My MacBook has gone to sleep. ZZZzzz???)
[6:17] * wschulze (~wschulze@cpe-69-206-251-158.nyc.res.rr.com) has joined #ceph
[6:22] * Pedras (~Adium@50.185.218.255) has joined #ceph
[6:23] * MACscr (~Adium@c-50-158-183-38.hsd1.il.comcast.net) Quit (Quit: Leaving.)
[6:30] * vbellur (~vijay@122.167.144.232) Quit (Read error: Connection reset by peer)
[6:34] * rongze_ (~rongze@114.54.30.94) has joined #ceph
[6:34] * rongze (~rongze@114.54.30.94) Quit (Read error: Connection reset by peer)
[6:34] * Pedras (~Adium@50.185.218.255) Quit (Quit: Leaving.)
[6:45] * vbellur (~vijay@122.178.201.180) has joined #ceph
[6:48] * ircolle-away (~Adium@2601:1:8380:2d9:2984:4c72:89a2:87f5) Quit (Quit: Leaving.)
[6:49] * wschulze (~wschulze@cpe-69-206-251-158.nyc.res.rr.com) Quit (Quit: Leaving.)
[6:51] * saurabh (~saurabh@121.244.87.117) has joined #ceph
[6:57] * rdas (~rdas@106.221.145.179) has joined #ceph
[7:09] * lalatenduM (~lalatendu@121.244.87.117) has joined #ceph
[7:13] * davidzlap (~Adium@ip68-4-173-198.oc.oc.cox.net) Quit (Quit: Leaving.)
[7:16] * lucas1 (~Thunderbi@222.240.148.154) has joined #ceph
[7:26] * MACscr (~Adium@c-50-158-183-38.hsd1.il.comcast.net) has joined #ceph
[7:27] <MACscr> lurbs: besides making up fake hosts in the crushmap, does any other type of networking have to be setup for those fake entries?
[7:29] * vbellur (~vijay@122.178.201.180) Quit (Ping timeout: 480 seconds)
[7:43] * farcaller (~farcaller@176.61.105.25) has joined #ceph
[7:45] <farcaller> hi all. I'm trying to build a test ceph cluster with ansible in vagrant. Following the docs, but ending up with a broken configuration. Config: https://gist.github.com/farcaller/de151908fee17b07c1e9
[7:45] <farcaller> health HEALTH_WARN 88 pgs degraded; 192 pgs stuck unclean
[7:45] <farcaller> any ideas?
[7:45] * madkiss (~madkiss@p5099fdaa.dip0.t-ipconnect.de) Quit (Quit: Leaving.)
[7:52] <iggy> farcaller: how many nodes? what version?
[8:00] <farcaller> iggy: firefly, one host, 1 mon + 2 osd
[8:00] <farcaller> running on ubuntu 14.04
[8:01] * vbellur (~vijay@121.244.87.124) has joined #ceph
[8:01] <iggy> good
[8:01] <iggy> so the default replication level changed recently from 2 to 3
[8:01] * theanalyst (~abhi@49.32.0.65) has joined #ceph
[8:01] <iggy> since you only have 2 OSDs, you obviously can't achieve repl=3
[8:02] <farcaller> iggy: I thought that osd pool default size = 2 deals with that, no?
[8:02] <iggy> change the default repl to 2, the repl of the existing pools to 2, and possibly reboot the OSDs
[8:02] <iggy> when did you set that?
[8:03] <farcaller> iggy: in the config: https://gist.github.com/farcaller/de151908fee17b07c1e9#file-ceph-conf-j2
[8:03] <iggy> but did you set that before you ever started the cluster? or after?
[8:03] <farcaller> before.
[8:04] <farcaller> # ceph osd pool get data size
[8:04] <farcaller> size: 2
[8:04] <farcaller> (same for metadata and rbd)
[8:04] <iggy> okay... then I got nothing
[8:05] <iggy> wait about 9 hours for the devs to be around
[8:05] * ikrstic (~ikrstic@109-92-191-74.dynamic.isp.telekom.rs) has joined #ceph
[8:06] <farcaller> ok, thanks
[8:08] <farcaller> also, any ideas why ceph-disk activate fails after ceph-disk prepare? After I run prepare, both osd daemons are up and running, so activate seems a bit redundant. Why is it in the docs?
[8:09] * fghaas (~florian@91-119-141-13.dynamic.xdsl-line.inode.at) has joined #ceph
[8:15] <loicd> FYI : http://www.claranet.fr/careers/jobs/hf-ecole-d%E2%80%99ing%C3%A9nieur-%E2%80%93-stage-stockage-ceph-%E2%80%93-bas%C3%A9-%C3%A0-rennes.html ( french )
[8:16] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[8:20] * fsimonce (~simon@host27-60-dynamic.26-79-r.retail.telecomitalia.it) has joined #ceph
[8:23] * reed (~reed@75-101-54-131.dsl.static.sonic.net) Quit (Ping timeout: 480 seconds)
[8:25] * zidarsk8 (~zidar@89-212-142-10.dynamic.t-2.net) has joined #ceph
[8:27] * zidarsk8 (~zidar@89-212-142-10.dynamic.t-2.net) has left #ceph
[8:35] * Cube (~Cube@66.87.66.1) has joined #ceph
[8:39] * aldavud (~aldavud@213.55.176.234) has joined #ceph
[8:44] * ikrstic (~ikrstic@109-92-191-74.dynamic.isp.telekom.rs) Quit (Quit: Konversation terminated!)
[8:44] * Nacer (~Nacer@c2s31-2-83-152-89-219.fbx.proxad.net) has joined #ceph
[8:44] * Cube (~Cube@66.87.66.1) Quit (Quit: Leaving.)
[8:45] * madkiss (~madkiss@p4FF5BE0A.dip0.t-ipconnect.de) has joined #ceph
[8:45] * Sysadmin88 (~IceChat77@94.4.20.0) Quit (Quit: Give a man a fish and he will eat for a day. Teach him how to fish, and he will sit in a boat and drink beer all day)
[8:45] * lucas1 (~Thunderbi@222.240.148.154) Quit (Quit: lucas1)
[8:48] * Hell_Fire_ (~HellFire@123-243-155-184.static.tpgi.com.au) has joined #ceph
[8:48] * b0e (~aledermue@juniper1.netways.de) has joined #ceph
[8:49] * Hell_Fire (~HellFire@123-243-155-184.static.tpgi.com.au) Quit (Read error: Connection reset by peer)
[8:51] * vbellur (~vijay@121.244.87.124) Quit (Ping timeout: 480 seconds)
[8:53] * Kioob`Taff (~plug-oliv@local.plusdinfo.com) has joined #ceph
[8:57] <farcaller> iggy: just for the note, I got 6 osds, still stuck at HEALTH_WARN 37 pgs degraded; 192 pgs stuck unclean :)
[8:57] * tty0 (~tty0@gate.netnerdz.se) has left #ceph
[9:02] * vbellur (~vijay@121.244.87.117) has joined #ceph
[9:06] * Nacer (~Nacer@c2s31-2-83-152-89-219.fbx.proxad.net) Quit (Remote host closed the connection)
[9:09] * BManojlovic (~steki@91.195.39.5) has joined #ceph
[9:09] * mjeanson (~mjeanson@00012705.user.oftc.net) Quit (Remote host closed the connection)
[9:11] * AfC (~andrew@nat-gw2.syd4.anchor.net.au) Quit (Quit: Leaving.)
[9:11] * rendar (~I@host231-182-dynamic.1-87-r.retail.telecomitalia.it) has joined #ceph
[9:13] * mjeanson (~mjeanson@bell.multivax.ca) has joined #ceph
[9:14] <singler> farcaller: are you still on one host? did you change failure domain to osd?
[9:16] <ssejourne> loicd: thanks. I had to do that by myself :)
[9:16] * thomnico (~thomnico@2a01:e35:8b41:120:5cea:c68b:73db:aa4f) has joined #ceph
[9:16] <loicd> ssejourne: :-)
[9:17] <farcaller> singler: still one host. but I've noticed that I get the ceph package from trusty-updates, so now trying to force it to install one from ceph repo
[9:17] <farcaller> I wonder if ubuntu's build has some issues
[9:17] * thomnico (~thomnico@2a01:e35:8b41:120:5cea:c68b:73db:aa4f) Quit (Remote host closed the connection)
[9:18] * thomnico (~thomnico@2a01:e35:8b41:120:5cea:c68b:73db:aa4f) has joined #ceph
[9:18] * thb (~me@2a02:2028:1c4:56e0:bcfb:7f76:bc48:904a) has joined #ceph
[9:19] * ikrstic (~ikrstic@c82-214-88-26.loc.akton.net) has joined #ceph
[9:19] <ssejourne> farcaller: try to use legacy tunables. It fixed the problem for me. But I'm still unable to understand why...
[9:19] <singler> farcaller: you need to change crush rules, because ceph wants to place replicas on other hosts
[9:21] <farcaller> singler: how do I do that?
[9:21] <ssejourne> farcaller: osd crush chooseleaf type = 0 instead of 1
[9:22] <farcaller> I was a bit confused by the docs saying '1. Typically a host containing one or more Ceph OSD Daemons'
[9:23] <singler> farcaller: I use this: http://ceph.com/docs/master/rados/operations/crush-map/?highlight=chooseleaf#editing-a-crush-map
[9:24] <singler> and chage host to osd
[9:25] <ssejourne> but '1' mean replicate based on host. if you dump your crush map, you'll see that type 1 is host, and type 0 is osd
[9:25] <farcaller> ah
[9:25] <farcaller> yes, osd crush chooseleaf type = 0 worked in the end
[9:25] <farcaller> thanks
[9:26] * garphy`aw is now known as garphy
[9:27] * steveeJ (~junky@client156.amh.kn.studentenwohnheim-bw.de) Quit (Remote host closed the connection)
[9:27] <farcaller> one more thing. If I have two drives /dev/sda and /dev/sdb, both with two partitions, and I have 4 osds on them, how do I make sure that it does replicate over disks, e.g. /dev/sda1 to /dev/sdb2, not /dev/sda1 to /dev/sda2?
[9:30] <singler> farcaller: it is recomended to use 1 osd per disk
[9:31] <singler> and for your question I guess you could modify crush, to have disk in tree and set it in cheeseleaf
[9:32] <singler> e.g, now crush has host -> osd, you could modify to have host -> disk -> osd
[9:33] <farcaller> thanks, I'll dig into that
[9:34] * lcavassa (~lcavassa@89.184.114.246) has joined #ceph
[9:34] * aldavud (~aldavud@213.55.176.234) Quit (Read error: Operation timed out)
[9:35] * rotbeard (~redbeard@b2b-94-79-138-170.unitymedia.biz) has joined #ceph
[9:39] * zidarsk8 (~zidar@2001:1470:fffd:101c:ea11:32ff:fe9a:870) has joined #ceph
[9:40] * cookednoodles (~eoin@eoin.clanslots.com) has joined #ceph
[9:41] * Midnightmyth (~quassel@93-167-84-102-static.dk.customer.tdc.net) has joined #ceph
[9:41] * zidarsk8 (~zidar@2001:1470:fffd:101c:ea11:32ff:fe9a:870) has left #ceph
[9:41] * steki (~steki@93-87-175-138.dynamic.isp.telekom.rs) has joined #ceph
[9:42] * BManojlovic (~steki@91.195.39.5) Quit (Ping timeout: 480 seconds)
[9:46] * Nacer (~Nacer@252-87-190-213.intermediasud.com) has joined #ceph
[9:47] <MACscr> hmm, i cant seem to get to create an rbd device image. It just hangs when i do "rbd create foo --size 4096 --pool platter -m 10.0.0.100 -k /etc/ceph/ceph.client.admin.keyring"
[9:48] * michalefty (~micha@p20030071CF58DA00A8F5E07D36BC2972.dip0.t-ipconnect.de) has joined #ceph
[9:48] * michalefty (~micha@p20030071CF58DA00A8F5E07D36BC2972.dip0.t-ipconnect.de) has left #ceph
[9:49] * ninkotech (~duplo@static-84-242-87-186.net.upcbroadband.cz) has joined #ceph
[9:49] * ninkotech__ (~duplo@static-84-242-87-186.net.upcbroadband.cz) Quit (Ping timeout: 480 seconds)
[9:56] * ScOut3R (~ScOut3R@catv-89-133-22-210.catv.broadband.hu) has joined #ceph
[10:02] * imriz (~imriz@82.81.163.130) has joined #ceph
[10:16] * TMM (~hp@sams-office-nat.tomtomgroup.com) has joined #ceph
[10:17] * jtaguinerd1 (~jtaguiner@112.205.2.112) has joined #ceph
[10:19] * jtaguinerd (~jtaguiner@125.212.121.243) Quit (Read error: Operation timed out)
[10:27] <tnt_> Am I the only one to notice that at each upgrade ceph needs more and more memory ... upgrading OSD to firefly and they take like 25% more memory than previously.
[10:28] <tnt_> same sort of thing happenned during the upgrade to emperor.
[10:28] * analbeard (~shw@support.memset.com) has joined #ceph
[10:36] <MACscr> hmm, any rbd command seems to hang
[10:36] <ghartz> tnt_, that's pretty huge
[10:36] <ghartz> (didn't check if it's true actually)
[10:38] <tnt_> ghartz: yes, I find it big too ... the OSD processes used to IDLE at around 800-900Mo not long ago and not it's like 1.2G.
[10:39] <ghartz> tnt_, this is the real memory used or virtually ?
[10:39] <ghartz> osd do a lot of cache
[10:39] <tnt_> ghartz: RES in top.
[10:40] <ghartz> so it seems legit
[10:41] <ghartz> ask someone from @ceph if it's normal
[10:41] <MACscr> hmm, im reading that the 3.2 kernel and ceph dont get along well and 3.8 and above is recommended. True?
[10:41] <singler> tnt_: checked one of my nodes top show RES about 601176
[10:42] <tnt_> singler: well, it's highly dependent on the number of PGs and objects afaik, so what's interesting to see is the usage over time.
[10:42] <tnt_> especially before / after an update.
[10:43] <singler> also I read somethere that after recovery process, OSD frees memory slowly, maybe your OSD didn't free all memory after recovery/rebalance?
[10:48] <tnt_> Maybe ... we'll see how it goes in the next few hours
[10:55] * b0e (~aledermue@juniper1.netways.de) Quit (Ping timeout: 480 seconds)
[10:58] * lucas1 (~Thunderbi@222.247.57.50) has joined #ceph
[10:59] * cronix1 (~cronix@5.199.139.166) has joined #ceph
[11:00] * zack_dolby (~textual@e0109-49-132-43-197.uqwimax.jp) Quit (Quit: My MacBook has gone to sleep. ZZZzzz???)
[11:07] <Jakey> https://www.irccloud.com/pastebin/y3roXlLi
[11:07] <Jakey> MACscr: i gettingthis error again
[11:07] <Jakey> don't know what the heckis wrong
[11:08] * cronix1 (~cronix@5.199.139.166) Quit (Ping timeout: 480 seconds)
[11:09] <MACscr> wth do you keep doing weird ass hostnames?
[11:10] <MACscr> you didnt listen to a single thing we we said
[11:16] * lucas1 (~Thunderbi@222.247.57.50) Quit (Quit: lucas1)
[11:17] * farcaller (~farcaller@176.61.105.25) Quit (Quit: leaving)
[11:33] * gillesMo (~gillesMo@00012912.user.oftc.net) has joined #ceph
[11:33] * csaesumu (~csaesumu@inf-205-58.inf.um.es) Quit (Remote host closed the connection)
[11:39] * daniel_bb (~daniel@soho73-234.sohonet.co.uk) Quit (Remote host closed the connection)
[11:44] * fghaas (~florian@91-119-141-13.dynamic.xdsl-line.inode.at) Quit (Quit: Leaving.)
[11:49] * rwheeler (~rwheeler@nat-pool-tlv-u.redhat.com) has joined #ceph
[11:55] * Vacum_ is now known as Vacum
[11:56] * lucas1 (~Thunderbi@222.240.148.130) has joined #ceph
[11:58] * b0e (~aledermue@juniper1.netways.de) has joined #ceph
[12:00] * jtaguinerd1 (~jtaguiner@112.205.2.112) Quit (Read error: Connection reset by peer)
[12:00] * jtaguinerd (~jtaguiner@112.205.2.112) has joined #ceph
[12:01] * thomnico (~thomnico@2a01:e35:8b41:120:5cea:c68b:73db:aa4f) Quit (Remote host closed the connection)
[12:04] * ScOut3R (~ScOut3R@catv-89-133-22-210.catv.broadband.hu) Quit (Read error: Operation timed out)
[12:09] * steki (~steki@93-87-175-138.dynamic.isp.telekom.rs) Quit (Read error: Operation timed out)
[12:10] * steki (~steki@91.195.39.5) has joined #ceph
[12:11] * theanalyst (~abhi@49.32.0.65) Quit (Read error: Connection reset by peer)
[12:13] * ScOut3R (~ScOut3R@catv-80-99-64-8.catv.broadband.hu) has joined #ceph
[12:14] * zhaochao (~zhaochao@124.207.139.17) has left #ceph
[12:14] * AfC (~andrew@2001:44b8:31cb:d400:6e88:14ff:fe33:2a9c) has joined #ceph
[12:22] * thomnico (~thomnico@2a01:e35:8b41:120:fc38:401b:c0da:8224) has joined #ceph
[12:41] * AfC (~andrew@2001:44b8:31cb:d400:6e88:14ff:fe33:2a9c) Quit (Quit: Leaving.)
[12:49] * boichev (~boichev@213.169.56.130) Quit (Quit: Nettalk6 - www.ntalk.de)
[12:50] * jtaguinerd (~jtaguiner@112.205.2.112) Quit (Read error: Connection reset by peer)
[12:51] * jtaguinerd (~jtaguiner@112.205.2.112) has joined #ceph
[12:53] * lucas1 (~Thunderbi@222.240.148.130) Quit (Quit: lucas1)
[12:57] * stewiem20001 (~stewiem20@195.10.250.233) has joined #ceph
[12:57] * stewiem2000 (~stewiem20@195.10.250.233) Quit (Read error: Connection reset by peer)
[12:57] * ufven (~ufven@130-229-29-136-dhcp.cmm.ki.se) Quit (Remote host closed the connection)
[12:58] * ufven (~ufven@130-229-29-136-dhcp.cmm.ki.se) has joined #ceph
[12:58] * danieljh (~daniel@0001b4e9.user.oftc.net) has joined #ceph
[12:58] * ninkotech (~duplo@static-84-242-87-186.net.upcbroadband.cz) Quit (Read error: Connection reset by peer)
[12:58] * ninkotech (~duplo@static-84-242-87-186.net.upcbroadband.cz) has joined #ceph
[13:08] * cronix1 (~cronix@5.199.139.166) has joined #ceph
[13:19] * analbeard (~shw@support.memset.com) Quit (Quit: Leaving.)
[13:19] * ninkotech (~duplo@static-84-242-87-186.net.upcbroadband.cz) Quit (Quit: Konversation terminated!)
[13:22] * rdas (~rdas@106.221.145.179) Quit (Quit: Leaving)
[13:26] * ninkotech (~duplo@static-84-242-87-186.net.upcbroadband.cz) has joined #ceph
[13:31] * singler (~singler@zeta.kirneh.eu) Quit (Quit: leaving)
[13:33] * saurabh (~saurabh@121.244.87.117) Quit (Quit: Leaving)
[13:36] * lalatenduM (~lalatendu@121.244.87.117) Quit (Quit: Leaving)
[13:36] * jtaguinerd (~jtaguiner@112.205.2.112) Quit (Read error: Connection reset by peer)
[13:47] * garphy is now known as garphy`aw
[13:47] * huangjun (~kvirc@111.174.239.37) Quit (Read error: Operation timed out)
[13:49] * garphy`aw is now known as garphy
[13:49] * lalatenduM (~lalatendu@121.244.87.124) has joined #ceph
[13:50] * rongze_ (~rongze@114.54.30.94) Quit (Remote host closed the connection)
[13:50] * fghaas (~florian@91-119-141-13.dynamic.xdsl-line.inode.at) has joined #ceph
[14:05] * vbellur (~vijay@121.244.87.117) Quit (Read error: Operation timed out)
[14:06] * zidarsk8 (~zidar@2001:1470:fffd:101c:ea11:32ff:fe9a:870) has joined #ceph
[14:18] * wschulze (~wschulze@cpe-69-206-251-158.nyc.res.rr.com) has joined #ceph
[14:21] * vbellur (~vijay@122.178.201.180) has joined #ceph
[14:25] * fsimonce is now known as fsimonce|afk
[14:25] * sz0_ (~sz0@46.197.48.116) Quit ()
[14:34] * zigo (quasselcor@ipv6-ftp.gplhost.com) Quit (Quit: No Ping reply in 180 seconds.)
[14:34] * zigo (quasselcor@ipv6-ftp.gplhost.com) has joined #ceph
[14:37] * wschulze (~wschulze@cpe-69-206-251-158.nyc.res.rr.com) Quit (Quit: Leaving.)
[14:38] * zigo (quasselcor@ipv6-ftp.gplhost.com) Quit ()
[14:39] * zigo (quasselcor@ipv6-ftp.gplhost.com) has joined #ceph
[14:43] * zigo (quasselcor@ipv6-ftp.gplhost.com) Quit ()
[14:45] * bkopilov (~bkopilov@nat-pool-tlv-t.redhat.com) Quit (Ping timeout: 480 seconds)
[14:45] * zigo (quasselcor@ipv6-ftp.gplhost.com) has joined #ceph
[14:48] <magicrobotmonkey> anyone seen this? agent_load_hit_sets: could not load hitset
[14:48] * shang (~ShangWu@42-68-246-231.EMOME-IP.hinet.net) has joined #ceph
[14:48] * zigo (quasselcor@ipv6-ftp.gplhost.com) Quit (Quit: No Ping reply in 180 seconds.)
[14:48] * zigo (quasselcor@ipv6-ftp.gplhost.com) has joined #ceph
[14:50] * danieljh (~daniel@0001b4e9.user.oftc.net) Quit (Quit: leaving)
[14:50] * jordanP (~jordan@2a04:2500:0:b00:50dd:8239:72fe:e90) has joined #ceph
[14:52] * zigo (quasselcor@ipv6-ftp.gplhost.com) Quit ()
[14:52] * zigo (quasselcor@ipv6-ftp.gplhost.com) has joined #ceph
[14:55] * primechuck (~primechuc@host-95-2-129.infobunker.com) has joined #ceph
[14:56] * zigo (quasselcor@ipv6-ftp.gplhost.com) Quit ()
[14:56] * zigo (quasselcor@ipv6-ftp.gplhost.com) has joined #ceph
[14:56] * primechu_ (~primechuc@host-95-2-129.infobunker.com) has joined #ceph
[14:57] * analbeard (~shw@support.memset.com) has joined #ceph
[14:59] * zigo (quasselcor@ipv6-ftp.gplhost.com) Quit ()
[15:00] * zigo (quasselcor@ipv6-ftp.gplhost.com) has joined #ceph
[15:01] * dvanders (~dvanders@2001:1458:202:180::102:f6c7) Quit (Read error: Connection reset by peer)
[15:02] * zidarsk8 (~zidar@2001:1470:fffd:101c:ea11:32ff:fe9a:870) Quit (Remote host closed the connection)
[15:03] * primechuck (~primechuc@host-95-2-129.infobunker.com) Quit (Ping timeout: 480 seconds)
[15:04] * zigo (quasselcor@ipv6-ftp.gplhost.com) Quit ()
[15:04] * zigo (quasselcor@ipv6-ftp.gplhost.com) has joined #ceph
[15:04] * wschulze (~wschulze@cpe-69-206-251-158.nyc.res.rr.com) has joined #ceph
[15:05] * koleosfuscus (~koleosfus@ws11-189.unine.ch) has joined #ceph
[15:06] * jordanP (~jordan@2a04:2500:0:b00:50dd:8239:72fe:e90) Quit (Read error: Connection timed out)
[15:07] * zigo (quasselcor@ipv6-ftp.gplhost.com) Quit ()
[15:07] * zigo (quasselcor@ipv6-ftp.gplhost.com) has joined #ceph
[15:08] * Midnightmyth (~quassel@93-167-84-102-static.dk.customer.tdc.net) Quit (Ping timeout: 480 seconds)
[15:11] * zigo (quasselcor@ipv6-ftp.gplhost.com) Quit ()
[15:12] * zigo (quasselcor@ipv6-ftp.gplhost.com) has joined #ceph
[15:13] * michalefty (~micha@p20030071CF539000A8F5E07D36BC2972.dip0.t-ipconnect.de) has joined #ceph
[15:16] * koleosfuscus (~koleosfus@ws11-189.unine.ch) Quit (Quit: koleosfuscus)
[15:17] * zigo (quasselcor@ipv6-ftp.gplhost.com) Quit (Quit: No Ping reply in 180 seconds.)
[15:17] * zigo (quasselcor@ipv6-ftp.gplhost.com) has joined #ceph
[15:21] * zigo (quasselcor@ipv6-ftp.gplhost.com) Quit ()
[15:21] * zigo (quasselcor@ipv6-ftp.gplhost.com) has joined #ceph
[15:22] * jordanP (~jordan@185.23.92.11) has joined #ceph
[15:22] * brad_mssw (~brad@shop.monetra.com) has joined #ceph
[15:23] * lalatenduM (~lalatendu@121.244.87.124) Quit (Quit: Leaving)
[15:26] * zigo (quasselcor@ipv6-ftp.gplhost.com) Quit ()
[15:26] * zigo (quasselcor@ipv6-ftp.gplhost.com) has joined #ceph
[15:29] * zigo (quasselcor@ipv6-ftp.gplhost.com) Quit ()
[15:30] * zigo (quasselcor@ipv6-ftp.gplhost.com) has joined #ceph
[15:33] * michalefty (~micha@p20030071CF539000A8F5E07D36BC2972.dip0.t-ipconnect.de) Quit (Quit: Leaving.)
[15:34] * zigo (quasselcor@ipv6-ftp.gplhost.com) Quit ()
[15:34] * zigo (quasselcor@ipv6-ftp.gplhost.com) has joined #ceph
[15:35] * koleosfuscus (~koleosfus@ws11-189.unine.ch) has joined #ceph
[15:38] * zigo (quasselcor@ipv6-ftp.gplhost.com) Quit ()
[15:38] * japuzzo (~japuzzo@pok2.bluebird.ibm.com) has joined #ceph
[15:38] * koleosfuscus (~koleosfus@ws11-189.unine.ch) Quit ()
[15:40] * Nacer (~Nacer@252-87-190-213.intermediasud.com) Quit (Remote host closed the connection)
[15:40] * zigo (quasselcor@ipv6-ftp.gplhost.com) has joined #ceph
[15:40] * Nacer (~Nacer@252-87-190-213.intermediasud.com) has joined #ceph
[15:40] * zigo (quasselcor@ipv6-ftp.gplhost.com) Quit (Killed (Myon (fix your bouncer, please)))
[15:40] * zigo (quasselcor@ipv6-ftp.gplhost.com) has joined #ceph
[15:44] * zigo (quasselcor@ipv6-ftp.gplhost.com) Quit ()
[15:44] * zigo (quasselcor@ipv6-ftp.gplhost.com) has joined #ceph
[15:45] * ana_ (~quassel@143.38.77.188.dynamic.jazztel.es) has joined #ceph
[15:45] * ana_ (~quassel@143.38.77.188.dynamic.jazztel.es) Quit (Remote host closed the connection)
[15:46] * zigo (quasselcor@ipv6-ftp.gplhost.com) Quit (autokilled: Please fix your bouncer (akilled for 8h) (2014-07-03 13:46:35))
[15:46] * madkiss (~madkiss@p4FF5BE0A.dip0.t-ipconnect.de) Quit (Quit: Leaving.)
[15:47] * fghaas (~florian@91-119-141-13.dynamic.xdsl-line.inode.at) has left #ceph
[15:48] * diegows (~diegows@190.190.5.238) has joined #ceph
[15:49] * Hell_Fire__ (~HellFire@123-243-155-184.static.tpgi.com.au) has joined #ceph
[15:49] * Hell_Fire_ (~HellFire@123-243-155-184.static.tpgi.com.au) Quit (Read error: Connection reset by peer)
[15:53] * ScOut3R_ (~ScOut3R@catv-80-99-64-8.catv.broadband.hu) has joined #ceph
[15:54] * scuttlemonkey is now known as scuttle|afk
[15:54] * allsystemsarego (~allsystem@86.121.85.84) has joined #ceph
[15:56] * ScOut3R (~ScOut3R@catv-80-99-64-8.catv.broadband.hu) Quit (Read error: Operation timed out)
[15:58] * huangjun (~kvirc@117.151.48.216) has joined #ceph
[16:01] * markbby (~Adium@168.94.245.1) has joined #ceph
[16:02] * primechu_ (~primechuc@host-95-2-129.infobunker.com) Quit (Remote host closed the connection)
[16:02] * primechuck (~primechuc@host-95-2-129.infobunker.com) has joined #ceph
[16:14] * thomnico (~thomnico@2a01:e35:8b41:120:fc38:401b:c0da:8224) Quit (Quit: Ex-Chat)
[16:19] * bkopilov (~bkopilov@213.57.17.156) has joined #ceph
[16:28] * analbeard (~shw@support.memset.com) Quit (Quit: Leaving.)
[16:29] * zigo_ (quasselcor@ipv6-ftp.gplhost.com) has joined #ceph
[16:30] * zigo_ (quasselcor@ipv6-ftp.gplhost.com) Quit ()
[16:30] * zigo (quasselcor@ipv6-ftp.gplhost.com) has joined #ceph
[16:34] <mo-> im seing lease_expire between mons indicating that the connection might be laggy. I know this is likely to be not a ceph issue, but how would one diagnose a laggy network (thats just on the backplane of a blade chassis, doesnt even involve any 'real' switch)? the network guys say theres no congestion going on. I can see ceph currently scrubbing (regular, not deep), but I dont think that could be it
[16:38] * allsystemsarego (~allsystem@86.121.85.84) Quit (Ping timeout: 480 seconds)
[16:39] * ikrstic (~ikrstic@c82-214-88-26.loc.akton.net) Quit (Quit: Konversation terminated!)
[16:40] * mkarnebeek (~mkarnebee@37.153.225.194) Quit ()
[16:40] * allsystemsarego (~allsystem@86.121.85.168) has joined #ceph
[16:40] <magicrobotmonkey> I've had a similar issue with one specific box, mo-
[16:40] <magicrobotmonkey> where ceph was acting like there were network problems but i had no other symptoms i could find
[16:40] <magicrobotmonkey> i havent sorted it yet, i just stopped using that box as a mon
[16:41] <mo-> weird
[16:41] <magicrobotmonkey> yea
[16:41] <magicrobotmonkey> im finding that ceph tends to find all the little problems with your infrastructure...
[16:42] <mo-> I remember people indicating that running different MTUs between nodes is likely to cause issues, but that doesnt seem the case here either
[16:42] * Nacer (~Nacer@252-87-190-213.intermediasud.com) Quit (Remote host closed the connection)
[16:42] * Nacer (~Nacer@252-87-190-213.intermediasud.com) has joined #ceph
[16:42] <mo-> thing is. it just started. has been running fine for a while...
[16:42] <magicrobotmonkey> yea im running mixed mtus on the rest of my hosts with no problems
[16:45] * Kioob`Taff (~plug-oliv@local.plusdinfo.com) Quit (Quit: Leaving.)
[16:47] <pressureman> mo-, mixed MTUs would only possibly cause problems with UDP traffic
[16:47] <mo-> yea but like I said, MTUs are all even here
[16:47] <pressureman> TCP will negotiate a MSS (Maximum Segment Size) based on the lower MTU of the two sides of the connection
[16:47] <mo-> I can high io waits on the ceph nodes though in top
[16:47] <mo-> wa ranges from 5 to 20
[16:48] <pressureman> most stuff that i've read says that jumbo frames are not worth the effort on Gig-E, but definitely make sense on 10GE
[16:48] <magicrobotmonkey> im getting that too on one node, mo-
[16:48] <magicrobotmonkey> what os are you on?
[16:48] <mo-> uh this is ubuntu 13.04 sadly (not my choice)
[16:49] <magicrobotmonkey> heh
[16:49] <magicrobotmonkey> well i have 3xosd nodes with ~28 osds each
[16:49] <magicrobotmonkey> two are ubuntu and they're running ok
[16:49] <mo-> 3 nodes, 2 OSDs each...
[16:49] <magicrobotmonkey> ones centos 6.5 and its being a real problem
[16:52] <mo-> wonder if I can cancel the scrubbing thats running
[16:56] <mo-> found something for that
[16:57] <absynth> if you set the scrub interval to something absurd, 99999999999 or so, it will stop scrubbing after completing the currently-scrubbing PG
[16:58] <mo-> I found "ceph osd set noscrub"
[17:00] * papisz (~pszablow@jutrzenka.firma.o2.pl) has joined #ceph
[17:02] <papisz> hi, I've got a strange error when trying to initiate the multi part upload on my local ceph instance
[17:03] <papisz> single part upload works just fine, but the multi part one gives me 403 Forbidden when I call the boto's initiate_multipart_upload method
[17:06] <papisz> any ideas? I thought it would be simple, there are many examples around, but none of them worked for me
[17:10] * steki (~steki@91.195.39.5) Quit (Ping timeout: 480 seconds)
[17:11] * KaZeR (~kazer@64.201.252.132) has joined #ceph
[17:19] * mattch (~mattch@pcw3047.see.ed.ac.uk) Quit (Remote host closed the connection)
[17:20] * thomnico (~thomnico@2a01:e35:8b41:120:fc38:401b:c0da:8224) has joined #ceph
[17:21] * mattch (~mattch@pcw3047.see.ed.ac.uk) has joined #ceph
[17:21] * b0e (~aledermue@juniper1.netways.de) Quit (Quit: Leaving.)
[17:24] * shang (~ShangWu@42-68-246-231.EMOME-IP.hinet.net) Quit (Quit: Ex-Chat)
[17:25] * wrencsok (~wrencsok@wsip-174-79-34-244.ph.ph.cox.net) has joined #ceph
[17:27] * ikrstic (~ikrstic@178-221-30-77.dynamic.isp.telekom.rs) has joined #ceph
[17:27] * analbeard (~shw@support.memset.com) has joined #ceph
[17:28] * reed (~reed@75-101-54-131.dsl.static.sonic.net) has joined #ceph
[17:30] * narb (~Jeff@38.99.52.10) has joined #ceph
[17:34] <dgarcia> Hi, I'm having trouble adding a rbd disk to a VM. I'm using Xen 4.4.0, libvirt, and ubuntu. I'm using this website (http://ceph.com/docs/master/rbd/libvirt/) to configure the VM but when I try to start the VM I get: error: Failed to start domain vm-01
[17:34] <dgarcia> error: internal error: libxenlight failed to create new domain 'vm-01'
[17:34] * imriz (~imriz@82.81.163.130) Quit (Ping timeout: 480 seconds)
[17:37] <dgarcia> does anyone have any ideas about this?
[17:45] * xarses (~andreww@c-24-23-183-44.hsd1.ca.comcast.net) Quit (Read error: Operation timed out)
[17:45] * TMM (~hp@sams-office-nat.tomtomgroup.com) Quit (Quit: Ex-Chat)
[17:46] * wrencsok (~wrencsok@wsip-174-79-34-244.ph.ph.cox.net) Quit (Quit: Leaving.)
[17:46] * Nacer_ (~Nacer@252-87-190-213.intermediasud.com) has joined #ceph
[17:46] * Nacer_ (~Nacer@252-87-190-213.intermediasud.com) Quit (Remote host closed the connection)
[17:47] * wrencsok (~wrencsok@wsip-174-79-34-244.ph.ph.cox.net) has joined #ceph
[17:47] * gillesMo (~gillesMo@00012912.user.oftc.net) Quit (Quit: Konversation terminated!)
[17:48] * Nacer (~Nacer@252-87-190-213.intermediasud.com) Quit (Read error: Operation timed out)
[17:54] * rwheeler (~rwheeler@nat-pool-tlv-u.redhat.com) Quit (Quit: Leaving)
[17:56] * Pedras (~Adium@50.185.218.255) has joined #ceph
[17:56] * Pedras (~Adium@50.185.218.255) Quit ()
[17:56] * Pedras (~Adium@216.207.42.140) has joined #ceph
[18:01] * angdraug (~angdraug@12.164.168.117) has joined #ceph
[18:01] * thb (~me@0001bd58.user.oftc.net) Quit (Ping timeout: 480 seconds)
[18:05] * xarses (~andreww@12.164.168.117) has joined #ceph
[18:11] * huangjun (~kvirc@117.151.48.216) Quit (Ping timeout: 480 seconds)
[18:12] * Pedras (~Adium@216.207.42.140) Quit (Quit: Leaving.)
[18:13] * Pedras (~Adium@216.207.42.140) has joined #ceph
[18:14] * hasues (~hasues@kwfw01.scrippsnetworksinteractive.com) has joined #ceph
[18:16] * joef (~Adium@2620:79:0:131:785f:b1d2:323e:95aa) has joined #ceph
[18:19] * sigsegv (~sigsegv@188.25.120.56) has joined #ceph
[18:21] * papisz (~pszablow@jutrzenka.firma.o2.pl) Quit (Ping timeout: 480 seconds)
[18:22] * xarses_ (~andreww@12.164.168.117) has joined #ceph
[18:25] * ScOut3R (~ScOut3R@catv-80-99-64-8.catv.broadband.hu) has joined #ceph
[18:25] * Sysadmin88 (~IceChat77@94.4.20.0) has joined #ceph
[18:31] * rotbeard (~redbeard@b2b-94-79-138-170.unitymedia.biz) Quit (Quit: Leaving)
[18:31] * ScOut3R_ (~ScOut3R@catv-80-99-64-8.catv.broadband.hu) Quit (Ping timeout: 480 seconds)
[18:31] * markbby (~Adium@168.94.245.1) Quit (Quit: Leaving.)
[18:35] <dgarcia> Hello everyone I'm trying to power on a VM with a rbd disk. Everything seems to be working ok but when I start the machine I'm getting:
[18:35] <dgarcia> #error: Failed to start domain vm-01
[18:35] <dgarcia> #error: internal error: libxenlight failed to create new domain 'vm-01'
[18:35] <dgarcia> This is the log generated when I'm trying to start the VM (/var/log/libvirt/libxl/vm-01.log) http://pastebin.com/KVZ064hW
[18:35] <dgarcia> does anyone has any ideas?
[18:37] * markbby (~Adium@168.94.245.1) has joined #ceph
[18:37] * garphy is now known as garphy`aw
[18:47] * ikrstic (~ikrstic@178-221-30-77.dynamic.isp.telekom.rs) Quit (Quit: Konversation terminated!)
[18:47] * danieljh (~daniel@0001b4e9.user.oftc.net) has joined #ceph
[18:52] * ikrstic (~ikrstic@178-221-30-77.dynamic.isp.telekom.rs) has joined #ceph
[18:57] <dgarcia> Nevermind, after several days I finally got it working. For those using virsh-qemu-xen. The XML on the website http://ceph.com/docs/master/rbd/libvirt/ is wrong.
[18:58] <dgarcia> On the line <source protocol='rbd' name='libvirt-pool/new-libvirt-image'> you need to add rbd: at the beginning of the name.
[18:58] * zerick (~eocrospom@190.187.21.53) has joined #ceph
[18:59] <dgarcia> the correct line of the parameter would be <source protocol='rbd' name='rbd:libvirt-pool/new-libvirt-image'>
[19:00] <dgarcia> not sure if this is a bug or I was missing something...
[19:00] <kraken> ???_???
[19:02] * zack_dolby (~textual@pdf8519e7.tokynt01.ap.so-net.ne.jp) has joined #ceph
[19:07] * zack_dolby (~textual@pdf8519e7.tokynt01.ap.so-net.ne.jp) Quit (Quit: My MacBook has gone to sleep. ZZZzzz???)
[19:07] * primechuck (~primechuc@host-95-2-129.infobunker.com) Quit (Remote host closed the connection)
[19:09] * sjusthm (~sam@66-214-251-229.dhcp.gldl.ca.charter.com) has joined #ceph
[19:10] * zack_dolby (~textual@pdf8519e7.tokynt01.ap.so-net.ne.jp) has joined #ceph
[19:11] * jordanP (~jordan@185.23.92.11) Quit (Quit: Leaving)
[19:11] * Sysadmin88 (~IceChat77@94.4.20.0) Quit (Quit: Light travels faster then sound, which is why some people appear bright, until you hear them speak)
[19:14] * wrencsok (~wrencsok@wsip-174-79-34-244.ph.ph.cox.net) Quit (Remote host closed the connection)
[19:16] * wrencsok (~wrencsok@wsip-174-79-34-244.ph.ph.cox.net) has joined #ceph
[19:16] * lcavassa (~lcavassa@89.184.114.246) Quit (Quit: Leaving)
[19:17] <sjusthm> joao: standup?
[19:20] * odyssey4me (~odyssey4m@165.233.71.2) has joined #ceph
[19:21] <MACscr> hmm, so should i be running Ubuntu Trusty versus Debian Wheezy because of the kernel limitations and issues with RBD, etc? I know I could use the backport repo for Wheezy, but I would like to still be able to use ksplice for automatic kernel updates (without rebooting) and they only support current stable kernels for OS's.
[19:22] <MACscr> dgarcia: thanks for the heads up
[19:23] * thomnico (~thomnico@2a01:e35:8b41:120:fc38:401b:c0da:8224) Quit (Quit: Ex-Chat)
[19:32] * Cube (~Cube@66-87-66-1.pools.spcsdns.net) has joined #ceph
[19:33] <cookednoodles> or you can compile your own
[19:33] <cookednoodles> its not exactly rocket science
[19:33] <MACscr> cookednoodles: not sure how that would solve my ability to use ksplice
[19:33] <cookednoodles> Well in an ideal world you wouldn't use ksplice
[19:34] <MACscr> in an ideal world i wouldnt need ksplice
[19:34] <cookednoodles> you'd just mark the osds as down, reboot the machine, and bingo
[19:34] <MACscr> but we dont live in such a world
[19:35] <MACscr> right, but this kernel issue would apply to all my rbd client hosts as well
[19:35] <cookednoodles> live migration :P
[19:35] <MACscr> so it just seems like it would be smarter to just run a more supported os
[19:36] <MACscr> cookednoodles: sure, but that of course just causes more support overhead
[19:37] <cookednoodles> I have it all automated
[19:37] <MACscr> your kernel patching with the live migration and reboots?
[19:37] <cookednoodles> I have a maintenance mode where it moves everything over, takes it down and reboots
[19:38] <cookednoodles> have your kernels on pxe netboot and its like 0 work
[19:39] * analbeard (~shw@support.memset.com) Quit (Quit: Leaving.)
[19:39] <MACscr> sure, but then you need to have a redundant pxe server as well
[19:39] <cookednoodles> well everything should be redundant
[19:41] <MACscr> i only have 10 total servers, so i cant get to redundant. Right now i have dual switches, 3 osd/mon hosts, a backup host (PXE is just a lxc guest), and 5 kvm hosts
[19:42] <cookednoodles> is ksplice actually supported for ubuntu server ?
[19:42] <MACscr> yep. ive been using it for about 4 or 5 years now with all my centos/ubuntu systems
[19:43] <cookednoodles> I thought it was just for desktop
[19:43] <cookednoodles> as its now just oracle linux
[19:43] <MACscr> maybe for non existing users before oracle took over
[19:44] <cookednoodles> ah, maybe legacy
[19:44] <cookednoodles> personally I couldn't trust that
[19:45] <MACscr> why? its a very respected process and used by a lot of people
[19:45] * zack_dolby (~textual@pdf8519e7.tokynt01.ap.so-net.ne.jp) Quit (Quit: My MacBook has gone to sleep. ZZZzzz???)
[19:47] <cookednoodles> well you risk them dropping it at some point
[19:48] <cookednoodles> anyway off track
[19:48] * andreask (~andreask@h081217017238.dyn.cm.kabsi.at) has joined #ceph
[19:48] * ChanServ sets mode +v andreask
[19:48] <cookednoodles> are your storage devices separate from the rest ? if so having a constantly updated kernel may not be so important
[19:53] * rturk|afk is now known as rturk
[19:54] * rotbeard (~redbeard@2a02:908:df19:9900:76f0:6dff:fe3b:994d) has joined #ceph
[19:54] * wrencsok (~wrencsok@wsip-174-79-34-244.ph.ph.cox.net) Quit (Quit: Leaving.)
[19:59] * scuttle|afk is now known as scuttlemonkey
[19:59] * zidarsk8 (~zidar@89-212-142-10.dynamic.t-2.net) has joined #ceph
[19:59] * wrencsok (~wrencsok@wsip-174-79-34-244.ph.ph.cox.net) has joined #ceph
[20:03] * ScOut3R (~ScOut3R@catv-80-99-64-8.catv.broadband.hu) Quit (Ping timeout: 480 seconds)
[20:05] * zidarsk8 (~zidar@89-212-142-10.dynamic.t-2.net) has left #ceph
[20:07] * nhm_ (~nhm@174-20-9-114.mpls.qwest.net) has joined #ceph
[20:08] * imriz (~imriz@5.29.200.177) has joined #ceph
[20:11] * rweeks (~rweeks@pat.hitachigst.com) has joined #ceph
[20:12] <MACscr> cookednoodles: they are on their own vlan, with no direct public ip's, but will be natted
[20:13] <MACscr> my servers are colo'ed
[20:13] * sputnik13 (~sputnik13@207.8.121.241) has joined #ceph
[20:13] * nhm (~nhm@184-97-129-14.mpls.qwest.net) Quit (Ping timeout: 480 seconds)
[20:13] * vanham (~vanham@mail.mav.com.br) has joined #ceph
[20:14] <vanham> Hello everyone! How are you all?
[20:16] <vanham> I'm having a performance problem here. The disks on my OSDs are basically idle'ing (2%-5% usage) but read/writing at my RBD's is too slow and my server using it is getting a load average in the hundreds. Ceph is 0.80.1-1~bpo70+1 on Debian Wheezy with kernel 3.11.6.
[20:17] <vanham> I use RBD for one partition (Apache's sites) and for a handful of VMs
[20:17] <vanham> On "iostat -xmt" I get OSD's with very low usage, but rbd's with 99%-100% usage.
[20:18] <vanham> My ceph -w status shows "2014-07-03 15:18:09.540633 mon.0 [INF] pgmap v27761785: 647 pgs: 647 active+clean; 1281 GB data, 3879 GB used, 8117 GB / 11997 GB avail; 3921 B/s rd, 25525 kB/s wr, 106 op/s"
[20:18] <vanham> So, everything should be "active+clean"
[20:18] <vanham> So everything is "active+clean", nothing is scrubbing or repairing
[20:21] <MACscr> what speed are you getting, on what network, what type of drives, and how many replicas?
[20:21] <vanham> Oh, btw, CPU usage is low, bellow 50%.
[20:21] <MACscr> also, how many monitors and osd's, etc
[20:22] <vanham> Network is GIG, OSD drives are SATA with SSD journal, 3 replicas for everything. 3 monitors, 12 OSD nodes.
[20:23] <MACscr> whats the network usage like?
[20:24] <MACscr> you arent going to get more than 30MB/s with that setup
[20:24] <vanham> Switches are Cisco
[20:24] <vanham> I could only dream that :)
[20:24] <vanham> Let me get that network usage here
[20:26] * davidzlap (~Adium@ip68-4-173-198.oc.oc.cox.net) has joined #ceph
[20:26] <vanham> Incoming rates: 162244.8 kbits/sec 104019.0 packets/sec - Outgoing rates: 178011.9 kbits/sec 86499.2 packets/sec
[20:27] <vanham> 200k packets per seconds seems like a lot...
[20:27] <iggy> is that on nodes hosting mons too?
[20:27] <iggy> because that wouldn't really surprise me
[20:28] <vanham> This is my first OSD node. Same node that mounts that RBD. Also my third mon
[20:28] <vanham> First and second mon are separate
[20:28] <vanham> Let me get that
[20:28] <MACscr> why would you be mounting rbd on your osd node?
[20:28] <iggy> you are mounting an RBD volume on an OSD?
[20:30] <vanham> mon.a: Incoming rates: 25549.7 kbits/sec 5577.2 packets/sec Outgoing rates: 50059.7 kbits/sec 4663.4 packets/sec
[20:30] <vanham> mon.b should be getting about that same amount of traffic
[20:30] <vanham> iggy, yes.
[20:30] <vanham> I'm not getting any memory starvation for now
[20:30] <vanham> kernel memory starvation
[20:31] <iggy> okay, but when it deadlocks you get to keep the pieces ;)
[20:31] <MACscr> vanham: why would you do that though? are your osd hosts also your web hosts?
[20:31] <vanham> 2 load balancers and 12 nodes
[20:31] <vanham> load balancers are also mon's
[20:31] <vanham> all 12 nodes do everything basically
[20:31] <MACscr> ah
[20:31] <iggy> oh wait, I read those numbers wrong...
[20:32] <iggy> are you sure you don't have some sort of network loop or something?
[20:32] <vanham> I used to have a more specialized server setup, but in the end we where wasting too much hardware, one would bottleneck on memory with CPU available, the other on CPU with memory available.
[20:33] <vanham> I moved down from 4 full racks to one now...
[20:33] <vanham> iggy, this should be about the amount of traffic I should be getting.
[20:33] <vanham> We only have one switch...
[20:34] <MACscr> ok, im running firefly on wheezy and cant seem to create my first rbd volume. I have at least upgraded my kernel on the rbd client host to 3.14, but when i try to create the volume, it just hangs.
[20:34] <MACscr> "rbd create foo --size 4096 --pool platter -m 10.0.0.100 -k /etc/ceph/ceph.client.admin.keyring"
[20:34] <vanham> I don't rule out a loop though
[20:34] <vanham> So, you both think this is network related?
[20:35] <MACscr> im not seeing any log data to help troubleshoot it yet. The verbosity is set to default
[20:35] <iggy> that just seems high unless there's more we don't know
[20:35] <vanham> It makes sense since CPU and Disk usage is not high...
[20:35] <iggy> have you done rados benches, etc. to try to narrow it down?
[20:35] <iggy> was this ever working properly?
[20:36] <vanham> Yeah, sure! But userbase is growing and I think I reached a bottleneck here. I just can't find it...
[20:37] <vanham> This is a dd for small writes: dd bs=1M count=1 if=/dev/zero of=test conv=fdatasync - 1048576 bytes (1.0 MB) copied, 0.683555 s, 1.5 MB/s
[20:37] <cookednoodles> could it be a single disk slowing it all down ?
[20:37] <cookednoodles> that sounds crazily slow
[20:37] <vanham> This is what is killing me: just tried to right 32k: 32768 bytes (33 kB) copied, 5.48576 s, 6.0 kB/s
[20:38] * aldavud (~aldavud@213.55.184.216) has joined #ceph
[20:38] <cookednoodles> have you looked at the ceph tools like calamari ?
[20:38] <iggy> what was the "Yeah, sure!" in response to? I asked a few questions
[20:39] <vanham> iggy, this was working before.
[20:39] <vanham> iggy, but as the user base is growing...
[20:39] <vanham> cookednoodles, "iostat -xmt 10" doesn't show anything greater than 8% for usage on any OSD
[20:40] <cookednoodles> that doesn't mean anything
[20:40] <vanham> iggy, Oh sorry, I didn't see the rados thing
[20:40] <vanham> iggy, no I haven't
[20:40] * andreask (~andreask@h081217017238.dyn.cm.kabsi.at) has left #ceph
[20:40] * Nacer (~Nacer@c2s31-2-83-152-89-219.fbx.proxad.net) has joined #ceph
[20:41] <vanham> cookednoodles, you know any command I could run to check for that?
[20:41] <cookednoodles> there are a ton of health commands, check the wiki
[20:42] <vanham> iggy, do you have a link on that rados benchmark?
[20:43] <iggy> just the docs
[20:43] <iggy> it's in there somewhere
[20:46] <vanham> Is there a problem if I run "rados bench" on a working production pool? I won't delete anything, right?
[20:46] <iggy> no
[20:47] * danieagle (~Daniel@179.184.165.184.static.gvt.net.br) Quit (Quit: Obrigado por Tudo! :-) inte+ :-))
[20:50] <vanham> rbd rados bench 60 write: Total time run:65.857780 Total writes made:559 Write size:4194304 Bandwidth (MB/sec):33.952 Stddev Bandwidth:20.9328 Max bandwidth (MB/sec):76 Min bandwidth (MB/sec):0 Average Latency:1.85574 Stddev Latency:1.54565 Max latency:7.50872 Min latency:0.097299
[20:50] <vanham> avg latency seems high
[20:51] <iggy> if that's seconds, yeah
[20:51] <iggy> also min bw 0...
[20:52] <vanham> It got stuck for a few seconds
[20:52] <iggy> I think there's another bench that will do the OSDs individually
[20:52] <iggy> you might have a bad disk somewhere
[20:53] <vanham> I usually check them all with smartctl every few weeks
[20:53] <cookednoodles> setup a monitor for that
[20:53] * denn1s (dennis@tilaa.krul.nu) has left #ceph
[20:53] <cookednoodles> no need to do stuff like that manually in 2014
[20:53] <iggy> calamari probably does that
[20:53] <iggy> if not, it should
[20:53] <cookednoodles> well collectd regardless
[20:53] <cookednoodles> you should have everything monitored
[20:54] * imriz (~imriz@5.29.200.177) Quit (Ping timeout: 480 seconds)
[20:54] <cookednoodles> hook it up to cabot and your full job is replaced with a script that'll email you when you have issues
[20:55] <vanham> I should have it on nagios already
[20:56] <vanham> we have about 450 tests on nagios, but not this one :)
[20:56] <cookednoodles> go do it :P
[20:56] <cookednoodles> there are probably some ceph bindings too
[20:56] <vanham> ceph is monitored! :)
[21:00] <vanham> Just ran a few dd on all OSDs: "1073741824 bytes (1.1 GB) copied, 7.85219 s, 137 MB/s"
[21:00] <vanham> This was an average result
[21:05] <MACscr> hmm, the default location for logs is /var/log/ceph, right? I dont seem to have any ceph logs in my log directory. Are there no logs generated by default or do i specifically have to enable them?
[21:05] <MACscr> trying to figure out why this rbd volume creation is just hanging
[21:08] * wer (~wer@206-248-239-142.unassigned.ntelos.net) Quit (Ping timeout: 480 seconds)
[21:08] * mtl2 (~Adium@c-67-174-109-212.hsd1.co.comcast.net) Quit (Quit: Leaving.)
[21:08] * scuttlemonkey is now known as scuttle|afk
[21:08] * scuttle|afk is now known as scuttlemonkey
[21:13] * rweeks (~rweeks@pat.hitachigst.com) Quit (Quit: Leaving)
[21:14] * odyssey4me (~odyssey4m@165.233.71.2) Quit (Quit: Leaving)
[21:15] <vanham> MACscr, take a look at https://ceph.com/docs/master/rados/troubleshooting/log-and-debug/
[21:15] <vanham> The boot time configs are easiest if you can restart your services
[21:16] <vanham> With ceph, logs can get *huge*
[21:17] * rendar (~I@host231-182-dynamic.1-87-r.retail.telecomitalia.it) Quit (Ping timeout: 480 seconds)
[21:20] * rendar (~I@host231-182-dynamic.1-87-r.retail.telecomitalia.it) has joined #ceph
[21:27] * colonD (~colonD@173-165-224-105-minnesota.hfc.comcastbusiness.net) Quit (Remote host closed the connection)
[21:31] * mjeanson (~mjeanson@00012705.user.oftc.net) Quit (Remote host closed the connection)
[21:33] * mjeanson (~mjeanson@bell.multivax.ca) has joined #ceph
[21:33] <MACscr> right, but shouldnt that just because of verbosity levels?
[21:38] * aldavud (~aldavud@213.55.184.216) Quit (Ping timeout: 480 seconds)
[21:44] * Nacer (~Nacer@c2s31-2-83-152-89-219.fbx.proxad.net) Quit (Remote host closed the connection)
[21:45] * allsystemsarego (~allsystem@86.121.85.168) Quit (Quit: Leaving)
[21:45] * MACscr (~Adium@c-50-158-183-38.hsd1.il.comcast.net) Quit (Ping timeout: 480 seconds)
[21:50] * MACscr (~Adium@c-50-158-183-38.hsd1.il.comcast.net) has joined #ceph
[21:50] * joshd (~joshd@38.122.20.226) Quit (Quit: Leaving.)
[21:58] * aldavud (~aldavud@217-162-119-191.dynamic.hispeed.ch) has joined #ceph
[22:01] * tws_ (~traviss@rrcs-24-123-86-154.central.biz.rr.com) Quit (Ping timeout: 480 seconds)
[22:01] * Nacer (~Nacer@c2s31-2-83-152-89-219.fbx.proxad.net) has joined #ceph
[22:02] * Sysadmin88 (~IceChat77@94.4.20.0) has joined #ceph
[22:10] * brad_mssw (~brad@shop.monetra.com) Quit (Quit: Leaving)
[22:16] <MACscr> is there a recommended default logging that you guys typically set?
[22:18] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[22:25] * `10 (~10@69.169.91.14) Quit (Remote host closed the connection)
[22:30] * cookednoodles (~eoin@eoin.clanslots.com) Quit (Quit: Ex-Chat)
[22:33] * aldavud (~aldavud@217-162-119-191.dynamic.hispeed.ch) Quit (Ping timeout: 480 seconds)
[22:35] * rturk is now known as rturk|afk
[22:40] * vanham (~vanham@mail.mav.com.br) Quit (Quit: Saindo)
[22:40] * Cube (~Cube@66-87-66-1.pools.spcsdns.net) Quit (Read error: Connection reset by peer)
[22:40] * Cube (~Cube@66-87-65-255.pools.spcsdns.net) has joined #ceph
[22:40] * wrencsok (~wrencsok@wsip-174-79-34-244.ph.ph.cox.net) Quit (Quit: Leaving.)
[22:41] <MACscr> hmm, actually any rbd command seems to hang, even info
[22:42] * ircolle (~Adium@mobile-166-147-083-056.mycingular.net) has joined #ceph
[22:42] * Nacer (~Nacer@c2s31-2-83-152-89-219.fbx.proxad.net) Quit (Read error: Connection reset by peer)
[22:42] * `10 (~10@69.169.91.14) has joined #ceph
[22:42] * wer (~wer@206-248-239-142.unassigned.ntelos.net) has joined #ceph
[22:43] * Nacer (~Nacer@c2s31-2-83-152-89-219.fbx.proxad.net) has joined #ceph
[22:44] * ircolle (~Adium@mobile-166-147-083-056.mycingular.net) Quit (Read error: Connection reset by peer)
[22:44] * fghaas1 (~florian@91-119-141-13.dynamic.xdsl-line.inode.at) has joined #ceph
[22:45] * ircolle (~Adium@mobile-166-147-083-056.mycingular.net) has joined #ceph
[22:45] * mnash (~chatzilla@vpn.expressionanalysis.com) Quit (Quit: ChatZilla 0.9.90.1 [Firefox 29.0.1/20140506152807])
[22:52] * sputnik13 (~sputnik13@207.8.121.241) Quit (Quit: Textual IRC Client: www.textualapp.com)
[22:53] * sputnik13 (~sputnik13@207.8.121.241) has joined #ceph
[22:56] * aldavud (~aldavud@217-162-119-191.dynamic.hispeed.ch) has joined #ceph
[22:58] * fghaas1 (~florian@91-119-141-13.dynamic.xdsl-line.inode.at) has left #ceph
[22:59] * ikrstic (~ikrstic@178-221-30-77.dynamic.isp.telekom.rs) Quit (Quit: Konversation terminated!)
[23:00] * japuzzo (~japuzzo@pok2.bluebird.ibm.com) Quit (Quit: Leaving)
[23:01] * rturk|afk is now known as rturk
[23:09] * Nacer (~Nacer@c2s31-2-83-152-89-219.fbx.proxad.net) Quit (Remote host closed the connection)
[23:11] * rendar (~I@host231-182-dynamic.1-87-r.retail.telecomitalia.it) Quit ()
[23:11] * joef1 (~Adium@2601:9:2a00:690:d51e:9073:e62d:e07e) has joined #ceph
[23:11] * joef1 (~Adium@2601:9:2a00:690:d51e:9073:e62d:e07e) has left #ceph
[23:12] * joef (~Adium@2620:79:0:131:785f:b1d2:323e:95aa) Quit (Remote host closed the connection)
[23:13] * joshd (~jdurgin@2607:f298:a:607:ac96:4f87:31e:c2d9) has joined #ceph
[23:14] * aldavud (~aldavud@217-162-119-191.dynamic.hispeed.ch) Quit (Read error: Operation timed out)
[23:18] * lpabon (~lpabon@nat-pool-bos-t.redhat.com) Quit (Read error: Connection reset by peer)
[23:33] * hasues (~hasues@kwfw01.scrippsnetworksinteractive.com) Quit (Quit: Leaving.)
[23:37] * sputnik13 (~sputnik13@207.8.121.241) Quit (Quit: My MacBook has gone to sleep. ZZZzzz???)
[23:39] * danieljh (~daniel@0001b4e9.user.oftc.net) Quit (Quit: leaving)
[23:44] * sputnik13 (~sputnik13@207.8.121.241) has joined #ceph
[23:44] * sputnik13 (~sputnik13@207.8.121.241) Quit ()
[23:49] * imjustmatthew (~imjustmat@pool-74-110-226-158.rcmdva.fios.verizon.net) Quit (Remote host closed the connection)
[23:49] * imjustmatthew (~imjustmat@pool-74-110-226-158.rcmdva.fios.verizon.net) has joined #ceph
[23:52] * analbeard (~shw@support.memset.com) has joined #ceph
[23:53] * sadbox (~jmcguire@sadbox.org) Quit (Quit: sadbox's ZNC may be broken)
[23:54] * sadbox (~jmcguire@sadbox.org) has joined #ceph
[23:54] * TMM (~hp@178-84-46-106.dynamic.upc.nl) has joined #ceph
[23:57] * LPG (~LPG@c-76-104-197-224.hsd1.wa.comcast.net) has joined #ceph

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.