#ceph IRC Log


IRC Log for 2013-09-19

Timestamps are in GMT/BST.

[0:00] * nwat (~nwat@c-50-131-197-174.hsd1.ca.comcast.net) has joined #ceph
[0:06] * sarob (~sarob@nat-dip28-wl-b.cfw-a-gci.corp.yahoo.com) has joined #ceph
[0:07] * nigwil_ (~chatzilla@2001:44b8:5144:7b00:25a3:b7f9:4449:6550) has joined #ceph
[0:07] * sarob (~sarob@nat-dip28-wl-b.cfw-a-gci.corp.yahoo.com) Quit (Remote host closed the connection)
[0:07] * sarob (~sarob@nat-dip4.cfw-a-gci.corp.yahoo.com) has joined #ceph
[0:08] * sprachgenerator (~sprachgen@ Quit (Quit: sprachgenerator)
[0:09] * ismell_ (~ismell@host-64-17-89-79.beyondbb.com) has joined #ceph
[0:09] * rendar (~s@host105-180-dynamic.1-87-r.retail.telecomitalia.it) Quit ()
[0:10] * ScOut3R (~scout3r@4E5C2305.dsl.pool.telekom.hu) has joined #ceph
[0:10] * ismell (~ismell@host-64-17-89-79.beyondbb.com) Quit (Ping timeout: 480 seconds)
[0:11] * sbadia (~sbadia@yasaw.net) has left #ceph
[0:11] * scuttlemonkey (~scuttlemo@ Quit (Ping timeout: 480 seconds)
[0:11] * sbadia_ (~sbadia@yasaw.net) has joined #ceph
[0:13] * nigwil (~chatzilla@2001:44b8:5144:7b00:281b:37d2:55ac:e71) Quit (Ping timeout: 480 seconds)
[0:14] * BManojlovic (~steki@fo-d- Quit (Quit: Ja odoh a vi sta 'ocete...)
[0:15] * nigwil_ (~chatzilla@2001:44b8:5144:7b00:25a3:b7f9:4449:6550) Quit (Ping timeout: 480 seconds)
[0:16] * mikedawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) has joined #ceph
[0:21] * nwat (~nwat@c-50-131-197-174.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[0:32] * Guest7042 (~a@ Quit (Quit: This computer has gone to sleep)
[0:32] * jcsp (~john@82-71-55-202.dsl.in-addr.zen.co.uk) Quit (Ping timeout: 480 seconds)
[0:33] * a (~a@ has joined #ceph
[0:34] * a is now known as Guest7108
[0:36] * sbadia_ is now known as sbadia
[0:37] * sjm (~sjm@dhcp-108-168-18-236.cable.user.start.ca) has joined #ceph
[0:41] * jcsp (~john@82-71-55-202.dsl.in-addr.zen.co.uk) has joined #ceph
[0:41] * malcolm (~malcolm@silico24.lnk.telstra.net) has joined #ceph
[0:46] * nhm (~nhm@ has joined #ceph
[0:46] * doxavore (~doug@99-7-52-88.lightspeed.rcsntx.sbcglobal.net) Quit (Quit: :qa!)
[0:48] * vata (~vata@2607:fad8:4:6:e963:8811:f4e6:44c0) Quit (Quit: Leaving.)
[0:48] * gregmark1 (~Adium@ Quit (Quit: Leaving.)
[0:50] * yanzheng (~zhyan@ has joined #ceph
[0:56] * sbadia (~sbadia@yasaw.net) Quit (Quit: Bye)
[0:57] * sbadia (~sbadia@yasaw.net) has joined #ceph
[0:58] * mtk (~mtk@ool-44c35983.dyn.optonline.net) Quit (Remote host closed the connection)
[1:00] * AfC (~andrew@2407:7800:200:1011:2ad2:44ff:fe08:a4c) has joined #ceph
[1:01] * ScOut3R (~scout3r@4E5C2305.dsl.pool.telekom.hu) Quit (Remote host closed the connection)
[1:08] * rturk is now known as rturk-away
[1:18] * dmsimard (~Adium@ Quit (Ping timeout: 480 seconds)
[1:19] * rturk-away is now known as rturk
[1:20] * jksM (~jks@3e6b5724.rev.stofanet.dk) Quit (Ping timeout: 480 seconds)
[1:24] * jeff-YF (~jeffyf@ Quit (Ping timeout: 480 seconds)
[1:26] * xmltok (~xmltok@cpe-76-170-26-114.socal.res.rr.com) has joined #ceph
[1:28] * nwat (~nwat@c-50-131-197-174.hsd1.ca.comcast.net) has joined #ceph
[1:31] * sbadia (~sbadia@yasaw.net) Quit (Quit: Bye)
[1:31] * sbadia (~sbadia@yasaw.net) has joined #ceph
[1:35] * diegows (~diegows@ Quit (Ping timeout: 480 seconds)
[1:39] * b1tbkt (~b1tbkt@24-217-192-155.dhcp.stls.mo.charter.com) Quit (Remote host closed the connection)
[1:42] * b1tbkt (~b1tbkt@24-217-192-155.dhcp.stls.mo.charter.com) has joined #ceph
[1:43] * bandrus1 (~Adium@cpe-76-95-217-129.socal.res.rr.com) Quit (Quit: Leaving.)
[1:47] * malcolm (~malcolm@silico24.lnk.telstra.net) Quit (Ping timeout: 480 seconds)
[1:49] * bandrus (~Adium@cpe-76-95-217-129.socal.res.rr.com) has joined #ceph
[1:55] * davidzlap (~Adium@cpe-75-84-249-188.socal.res.rr.com) has joined #ceph
[2:00] * AfC (~andrew@2407:7800:200:1011:2ad2:44ff:fe08:a4c) Quit (Ping timeout: 480 seconds)
[2:03] * Guest7108 (~a@ Quit (Quit: This computer has gone to sleep)
[2:12] * nwat (~nwat@c-50-131-197-174.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[2:14] * sarob_ (~sarob@nat-dip28-wl-b.cfw-a-gci.corp.yahoo.com) has joined #ceph
[2:14] * eternaleye (~eternaley@2002:3284:29cb::1) Quit (Ping timeout: 480 seconds)
[2:17] * sarob_ (~sarob@nat-dip28-wl-b.cfw-a-gci.corp.yahoo.com) Quit (Read error: Operation timed out)
[2:18] * andreask (~andreask@h081217135028.dyn.cm.kabsi.at) Quit (Ping timeout: 480 seconds)
[2:21] * malcolm (~malcolm@linux-549s.qut.edu.au) has joined #ceph
[2:21] * sarob (~sarob@nat-dip4.cfw-a-gci.corp.yahoo.com) Quit (Ping timeout: 480 seconds)
[2:24] * buck (~buck@c-24-6-91-4.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[2:25] * jlogan (~Thunderbi@2600:c00:3010:1:1::40) Quit (Quit: jlogan)
[2:29] * scuttlemonkey (~scuttlemo@ has joined #ceph
[2:29] * ChanServ sets mode +o scuttlemonkey
[2:32] * matsuhashi (~matsuhash@124x35x46x7.ap124.ftth.ucom.ne.jp) has joined #ceph
[2:33] * eternaleye (~eternaley@c-50-132-41-203.hsd1.wa.comcast.net) has joined #ceph
[2:33] * jlogan (~Thunderbi@2600:c00:3010:1:1::40) has joined #ceph
[2:39] * nwat (~nwat@c-24-5-146-110.hsd1.ca.comcast.net) has joined #ceph
[2:41] * b1tbkt (~b1tbkt@24-217-192-155.dhcp.stls.mo.charter.com) Quit (Remote host closed the connection)
[2:45] * b1tbkt (~b1tbkt@24-217-192-155.dhcp.stls.mo.charter.com) has joined #ceph
[2:51] * KevinPerks (~Adium@cpe-066-026-252-218.triad.res.rr.com) Quit (Quit: Leaving.)
[2:51] * KevinPerks (~Adium@cpe-066-026-252-218.triad.res.rr.com) has joined #ceph
[2:52] * xarses (~andreww@ Quit (Ping timeout: 480 seconds)
[2:53] * malcolm (~malcolm@linux-549s.qut.edu.au) Quit (Ping timeout: 480 seconds)
[2:53] * eternaleye (~eternaley@c-50-132-41-203.hsd1.wa.comcast.net) Quit (Remote host closed the connection)
[2:53] * eternaleye (~eternaley@c-50-132-41-203.hsd1.wa.comcast.net) has joined #ceph
[3:03] * mnash (~chatzilla@vpn.expressionanalysis.com) Quit (Ping timeout: 480 seconds)
[3:07] * glzhao (~glzhao@ has joined #ceph
[3:07] * WebSpider (~webspider@2001:7b8:1518:0:6100:9a60:360:62c0) has joined #ceph
[3:08] <WebSpider> hi! is anyone aware of a succesfull install for ceph as a tor hidden service?
[3:08] * sjm (~sjm@dhcp-108-168-18-236.cable.user.start.ca) Quit (Read error: Connection reset by peer)
[3:15] * b1tbkt (~b1tbkt@24-217-192-155.dhcp.stls.mo.charter.com) Quit (Remote host closed the connection)
[3:16] * Vjarjadian (~IceChat77@ Quit (Ping timeout: 480 seconds)
[3:20] * LeaChim (~LeaChim@host86-135-252-168.range86-135.btcentralplus.com) Quit (Remote host closed the connection)
[3:20] * xarses (~andreww@c-71-202-167-197.hsd1.ca.comcast.net) has joined #ceph
[3:23] * eternaleye (~eternaley@c-50-132-41-203.hsd1.wa.comcast.net) Quit (Ping timeout: 480 seconds)
[3:26] * kraken (~kraken@c-24-131-46-23.hsd1.ga.comcast.net) Quit (Ping timeout: 480 seconds)
[3:29] * mnash (~chatzilla@66-194-114-178.static.twtelecom.net) has joined #ceph
[3:32] * mikedawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) Quit (Read error: Connection reset by peer)
[3:37] * bandrus (~Adium@cpe-76-95-217-129.socal.res.rr.com) Quit (Quit: Leaving.)
[3:52] * angdraug (~angdraug@ Quit (Quit: Leaving)
[4:00] * athrift_ (~nz_monkey@ has joined #ceph
[4:03] * athrift (~nz_monkey@ Quit (Ping timeout: 480 seconds)
[4:04] * rturk is now known as rturk-away
[4:05] * Cube (~Cube@cpe-76-95-217-129.socal.res.rr.com) Quit (Quit: Leaving.)
[4:07] * xiaoxi (~xiaoxi@ has joined #ceph
[4:08] <xiaoxi> hi ,man
[4:16] * AfC (~andrew@ has joined #ceph
[4:26] * ircolle (~Adium@c-67-165-237-235.hsd1.co.comcast.net) Quit (Quit: Leaving.)
[4:30] <Qu310> anyone around using Dell R515's with ceph?
[4:30] <dlan> xiaoxi: hi, today is holiday, and you still working? ^-^
[4:31] <dlan> Qu310: what's the problem, never thought ceph rely on specific hardware..
[4:31] <Qu310> dlan: ah just trying to findout if the performance i'm getting is expected or not
[4:32] <Qu310> dlan: With the particular make up of the R515's using the sas expanders etc
[4:40] * nwat (~nwat@c-24-5-146-110.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[4:43] * ircolle (~Adium@c-67-165-237-235.hsd1.co.comcast.net) has joined #ceph
[4:43] * glzhao (~glzhao@ Quit (Quit: leaving)
[4:45] * sarob (~sarob@c-50-161-65-119.hsd1.ca.comcast.net) has joined #ceph
[4:46] * angdraug (~angdraug@c-98-248-39-148.hsd1.ca.comcast.net) has joined #ceph
[4:49] * nwat (~nwat@c-50-131-197-174.hsd1.ca.comcast.net) has joined #ceph
[4:54] <xiaoxi> dlan: basically not...but just nothing to do
[4:55] <xiaoxi> Qu310:could you explain more about your problem? eg:HW configuration(we are not that familiar with module name), performance you got, your expectaion
[4:55] * sagelap (~sage@ has joined #ceph
[4:59] * KevinPerks (~Adium@cpe-066-026-252-218.triad.res.rr.com) Quit (Quit: Leaving.)
[5:05] * fireD_ (~fireD@93-139-128-199.adsl.net.t-com.hr) has joined #ceph
[5:05] <Qu310> xiaoxi: sorry was afk for a sec
[5:07] * fireD (~fireD@93-139-176-126.adsl.net.t-com.hr) Quit (Ping timeout: 480 seconds)
[5:07] <Qu310> xiaoxi: basically we have 3 R515's with 12 disks each, and i'm getting pretty boring performance, in that the disk io seems ok, when i do a dd to file using oflag=direct i get about 60~mb/s write per disk and 125-140mb/s all running at the same time so the bus has enough bandwidth, but when doing a rados bench i only get around 400mb/s
[5:07] <Qu310> and if i run the bench on 3 nodes at once the 400 drops to 100-200 tops
[5:08] <Qu310> i have dual 10gbe running on each node and using iperf i get around 16gbit/s
[5:08] * nwat (~nwat@c-50-131-197-174.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[5:08] <Qu310> so seems kind of odd the numbers i'm getting
[5:09] <Qu310> i was using some SSD's however i stuffed up and only got the 100gb intel dc's which are only good for around 200mb/s tops
[5:10] <Qu310> so i took them out of the equasion and using direct disk journals which is a 2nd partition on the same physical disk as the osd
[5:12] <xiaoxi> sorry but what do you mean by 60MB/S per disk and 125~140MB/s for "all running at the same time" ? do you mean the aggregation BW of 12 disks only 125~140MB/s?
[5:12] <Qu310> sorry
[5:13] <Qu310> i ment, aggregation bw of the 12 disks is in excess of 1500mb/s read
[5:13] <Qu310> but write should be simular raw
[5:13] <Qu310> however when writing to a file clearly its going to be half of that again (journaling fs)
[5:14] <xiaoxi> so you have 60MB/s for write and 120MB/s for read on raw disk ?
[5:14] <Qu310> yerp
[5:14] <Qu310> rather
[5:14] <Qu310> not on 'raw" but on xfs
[5:14] <Qu310> for clarity raw would be /dev/sdb for example
[5:15] <Qu310> and i'm talking about a mounted disk ie : /dev/sdb mounted as /mnt/sdb1
[5:15] <Qu310> using xfs as the file system
[5:15] <xiaoxi> OK, so you don't use SSD as journal? I am not really follow why 200MB/s is not ideal for you ? is that due to you only have one ssd?
[5:15] <Qu310> i had 2 ssd's however they were really slow for journaling for some reason
[5:16] <Qu310> is specifications said around 200mb/s write however i was seeing less then that
[5:16] <Qu310> each
[5:16] <Qu310> so i've taken the ssd's out of it for the moment and just using a raw parition on the same disk as the osd
[5:17] <Qu310> for journaling
[5:18] <Qu310> so i suppose the question is
[5:18] <xiaoxi> may be due to the HBA/RAID card limitation. OK, So since you use second partition as journal , your expectation should be (60*12*3)/2(replica)/2( journal) =540MB/s
[5:18] <xiaoxi> for write
[5:19] <xiaoxi> you got 400, I think it's good since it's not sequential at all, the spindle had to seek back and forth between journal and datadisk
[5:19] * glzhao (~glzhao@ has joined #ceph
[5:20] <Qu310> true
[5:21] <Qu310> so i guess the bottleneck here then is the lack of ssd's
[5:21] * cmdrk writes this stuff down
[5:21] <Qu310> or something other then the osd to write the journal to
[5:21] <cmdrk> just bought 3 R510s myself :)
[5:21] <cmdrk> what kind of disks are in your R515's ?
[5:21] <Qu310> cmdrk: pretty good bang for buck i thought =)
[5:22] <Qu310> 3TB 7200 SAS
[5:22] <Qu310> well they have a SAS interfaces O.o
[5:22] <cmdrk> ya
[5:22] <xiaoxi> basically the access pattern on data disk is not that sequential even you move journal out of the disk, what I typically expect is ~50MB/s(with 120MB+/s for raw disk performance)
[5:23] <cmdrk> i ended up with 4TB disks and some SSDs .. we'll see how performance turns out
[5:23] <Qu310> cmdrk: which ssd's did you end up with?
[5:23] <cmdrk> the 200GB intel ones
[5:23] <Qu310> xiaoxi: so around 50mb/s read even using rados bench?
[5:23] <xiaoxi> I mean 50MB/s for both R/W using rados
[5:24] <Qu310> cmdrk: at least you didnt get caught like i did with the 100gb ones
[5:24] <cmdrk> are the 100s bad ?
[5:24] <xiaoxi> for read, an important trick may be you should set the readahead_kb to more than 512, personally I think 2048 is good
[5:24] <Qu310> xiaoxi: per osd? so upto 600mb/s read/write
[5:25] <Qu310> cmdrk: issue i have is the 100's are only good for 200mb/s tops
[5:25] <xiaoxi> 600MB for write, ~1000 for read
[5:25] <Qu310> other issue i have is when using the SSD's my performance tops out at like 100 odd mb/s
[5:25] <xiaoxi> what kind of 100GB SSD you have?
[5:25] <Qu310> intel DC3200's?
[5:26] <cmdrk> ahhh. im guessing thats due to the density of the chip layout ? i think generally more chips in parallel -> faster SSDs?
[5:26] <xiaoxi> well...that should be very old one
[5:26] <Qu310> sorry 3700's
[5:26] <Qu310> Sequential Read 500 MB/s
[5:26] <Qu310> Sequential Write 200 MB/s
[5:27] <Qu310> Random Read (100% Span) 75000 IOPS
[5:27] <Qu310> Random Write (100% Span)19000 IOPS
[5:27] <Qu310> my eyes are painted on i sware
[5:28] * glzhao_ (~glzhao@ has joined #ceph
[5:28] <xiaoxi> yes, 100GB is not a good choice
[5:28] * malcolm (~malcolm@silico24.lnk.telstra.net) has joined #ceph
[5:28] <xiaoxi> 200GB 3700 is good
[5:28] <Qu310> xiaoxi: read ahead on?
[5:28] <xiaoxi> could have ~400MB/s for write
[5:29] <xiaoxi> echo "2048" > /sys/block/{sdx}/queue/readahead_kb
[5:29] <Qu310> yeah they are allmost double in everyway :)
[5:30] <xiaoxi> yes, but 200GB is almost the same with 400GB, since ceph do not require large capacity, so 200GB shoud be the best choice
[5:30] * glzhao_ (~glzhao@ Quit ()
[5:30] <lurbs> I had this argument recently. "Yes, we're only going to be using around 60 GB of space for journals, but we need the 200 GB drives because they're much faster for writes."
[5:30] <xiaoxi> true
[5:30] <Qu310> yeah hindsights a pain sometimes
[5:31] <lurbs> Put the 100's in your workstation and buy more. ;)
[5:31] <Qu310> hehe how tempting it is
[5:31] <xiaoxi> but I think it's make sense, you may know that if you overcommitment your SSD(aka, only use < 80% of it's capacity) ,you will get better performance and longer durability
[5:32] <Qu310> xiaoxi: for sure
[5:32] <xiaoxi> go to buy more 3700 from us,haha
[5:33] <Qu310> xiaoxi: is the bufffer readahead setting persistant?
[5:33] <xiaoxi> no, reboot will reset the value to default ,which is 128K and will definitely kill your performance
[5:33] <Qu310> no worries
[5:33] <lurbs> xiaoxi: Yeah, plus those spare area benefits. Although the 200GB S3700's already have something pretty ridiculous like 260 GB of NAND onboard.
[5:36] <xiaoxi> Basically ,Intel has one extra NAND for XOR( you can image it as inter-NAND-RAID5) , and another NAND for over-commitment
[5:37] * sarob (~sarob@c-50-161-65-119.hsd1.ca.comcast.net) Quit (Remote host closed the connection)
[5:37] * sarob (~sarob@2601:9:7080:13a:c4d2:6087:6cc8:d76) has joined #ceph
[5:37] * Cube (~Cube@66-87-64-12.pools.spcsdns.net) has joined #ceph
[5:45] * sarob (~sarob@2601:9:7080:13a:c4d2:6087:6cc8:d76) Quit (Ping timeout: 480 seconds)
[5:48] * sarob (~sarob@c-50-161-65-119.hsd1.ca.comcast.net) has joined #ceph
[6:01] * xarses (~andreww@c-71-202-167-197.hsd1.ca.comcast.net) Quit (Remote host closed the connection)
[6:06] * wschulze (~wschulze@cpe-72-229-37-201.nyc.res.rr.com) has joined #ceph
[6:06] * sarob (~sarob@c-50-161-65-119.hsd1.ca.comcast.net) Quit (Remote host closed the connection)
[6:08] <Qu310> when using a public & cluster network, how do you let ceph know were the nodes are on the cluster network vs the public net? is it just dns based?
[6:08] * eternaleye_ (~eternaley@c-50-132-41-203.hsd1.wa.comcast.net) has joined #ceph
[6:09] <Qu310> if so why have the public network and cluster network in the ceph.conf?
[6:10] * eternaleye (~eternaley@c-50-132-41-203.hsd1.wa.comcast.net) has joined #ceph
[6:34] * ikla_ (~lbz@c-67-190-136-245.hsd1.co.comcast.net) Quit (Quit: Lost terminal)
[6:36] * yehuda_hm (~yehuda@99-48-177-65.lightspeed.irvnca.sbcglobal.net) Quit (Remote host closed the connection)
[6:37] <xiaoxi> Qu310: Public network is used for OSD<-> Client traffic
[6:37] <xiaoxi> Cluster network is used for OSD<-> OSD traffic
[6:38] * glzhao (~glzhao@ Quit (Quit: leaving)
[6:38] <xiaoxi> so, different kinds of messages are send out via different socket fds
[6:38] <xiaoxi> thus go to different network
[6:39] * a (~a@pool-173-55-143-200.lsanca.fios.verizon.net) has joined #ceph
[6:40] * a is now known as Guest7134
[6:58] * wschulze (~wschulze@cpe-72-229-37-201.nyc.res.rr.com) Quit (Quit: Leaving.)
[6:59] * Guest7134 (~a@pool-173-55-143-200.lsanca.fios.verizon.net) Quit (Quit: This computer has gone to sleep)
[7:02] * scuttlemonkey (~scuttlemo@ Quit (Ping timeout: 480 seconds)
[7:05] * xmltok (~xmltok@cpe-76-170-26-114.socal.res.rr.com) Quit (Quit: Bye!)
[7:13] * scuttlemonkey (~scuttlemo@ has joined #ceph
[7:13] * ChanServ sets mode +o scuttlemonkey
[7:24] * Kioob`Taff (~plug-oliv@local.plusdinfo.com) Quit (Ping timeout: 480 seconds)
[7:30] * Vjarjadian (~IceChat77@ has joined #ceph
[7:32] * a_ (~a@pool-173-55-143-200.lsanca.fios.verizon.net) has joined #ceph
[7:37] * grepory (~Adium@c-69-181-42-170.hsd1.ca.comcast.net) has joined #ceph
[7:42] * davidzlap (~Adium@cpe-75-84-249-188.socal.res.rr.com) Quit (Quit: Leaving.)
[7:42] * grepory (~Adium@c-69-181-42-170.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[7:43] * bandrus (~Adium@cpe-76-95-217-129.socal.res.rr.com) has joined #ceph
[7:48] * ircolle (~Adium@c-67-165-237-235.hsd1.co.comcast.net) Quit (Quit: Leaving.)
[7:51] <Qu310> xiaoxi: so how do you tell ceph were the nodes are located on each of the networks tho?
[7:56] * scuttlemonkey (~scuttlemo@ Quit (Ping timeout: 480 seconds)
[7:56] * yanzheng (~zhyan@ Quit (Remote host closed the connection)
[8:08] * yanzheng (~zhyan@jfdmzpr01-ext.jf.intel.com) has joined #ceph
[8:09] * topro (~topro@host-62-245-142-50.customer.m-online.net) has joined #ceph
[8:12] * a_ (~a@pool-173-55-143-200.lsanca.fios.verizon.net) Quit (Quit: This computer has gone to sleep)
[8:21] * angdraug (~angdraug@c-98-248-39-148.hsd1.ca.comcast.net) Quit (Quit: Leaving)
[8:28] * Vjarjadian (~IceChat77@ Quit (Quit: If you think nobody cares, try missing a few payments)
[8:37] * nigwil (~chatzilla@2001:44b8:5144:7b00:dc7a:214e:dc79:ee1) has joined #ceph
[8:38] * peetaur (~peter@CPEbc1401e60493-CMbc1401e60490.cpe.net.cable.rogers.com) Quit (Ping timeout: 480 seconds)
[8:42] * mattt (~mattt@ has joined #ceph
[8:51] * gucki (~smuxi@HSI-KBW-109-192-187-143.hsi6.kabel-badenwuerttemberg.de) has joined #ceph
[8:55] * jcsp (~john@82-71-55-202.dsl.in-addr.zen.co.uk) Quit (Read error: Connection reset by peer)
[8:56] * jcsp (~john@82-71-55-202.dsl.in-addr.zen.co.uk) has joined #ceph
[9:11] * odyssey4me (~odyssey4m@ has joined #ceph
[9:12] * JustEra (~JustEra@ has joined #ceph
[9:13] * yanzheng (~zhyan@jfdmzpr01-ext.jf.intel.com) Quit (Remote host closed the connection)
[9:15] * malcolm (~malcolm@silico24.lnk.telstra.net) Quit (Read error: Operation timed out)
[9:20] * allsystemsarego (~allsystem@ has joined #ceph
[9:26] * LeaChim (~LeaChim@host86-135-252-168.range86-135.btcentralplus.com) has joined #ceph
[9:27] * pieter (~pieter@105-236-215-222.access.mtnbusiness.co.za) has joined #ceph
[9:28] <pieter> Hi guys. Doing an rsync to my ceph cluster via writing to cephfs I started getting the following alerts at night: HEALTH_WARN 288 requests are blocked > 32 sec
[9:28] <pieter> Is this due to slow osds?
[9:29] * andreask (~andreask@h081217135028.dyn.cm.kabsi.at) has joined #ceph
[9:29] * ChanServ sets mode +v andreask
[9:29] <pieter> (It's 2x 2TB SATA XFS each with 1x 1TB SATA for Journal)
[9:33] * BManojlovic (~steki@ has joined #ceph
[9:44] * AfC (~andrew@ Quit (Read error: Operation timed out)
[9:49] * ScOut3R (~ScOut3R@catv-89-133-25-52.catv.broadband.hu) has joined #ceph
[9:50] * bandrus (~Adium@cpe-76-95-217-129.socal.res.rr.com) Quit (Quit: Leaving.)
[9:51] <pieter> Also, would adding 2x more SATA OSD's improve speed? (Though they won't have separate journals)
[9:55] * schlitzer|work (~schlitzer@ has joined #ceph
[10:03] * bandrus (~Adium@cpe-76-95-217-129.socal.res.rr.com) has joined #ceph
[10:07] * JustEra (~JustEra@ Quit (Ping timeout: 480 seconds)
[10:08] * JustEra (~JustEra@ has joined #ceph
[10:14] * bandrus (~Adium@cpe-76-95-217-129.socal.res.rr.com) Quit (Quit: Leaving.)
[10:19] * thomnico (~thomnico@ has joined #ceph
[10:20] * ismell (~ismell@host-64-17-89-79.beyondbb.com) has joined #ceph
[10:20] * ismell_ (~ismell@host-64-17-89-79.beyondbb.com) Quit (Read error: Connection reset by peer)
[10:23] * Kioob`Taff (~plug-oliv@local.plusdinfo.com) has joined #ceph
[10:25] * ismell_ (~ismell@host-64-17-89-79.beyondbb.com) has joined #ceph
[10:25] * ismell (~ismell@host-64-17-89-79.beyondbb.com) Quit (Read error: Connection reset by peer)
[10:26] * kleini (~marcus@2001:4dd0:ff00:88c0:f66d:4ff:fe15:b1c3) has joined #ceph
[10:27] * bandrus (~Adium@cpe-76-95-217-129.socal.res.rr.com) has joined #ceph
[10:27] * matsuhashi (~matsuhash@124x35x46x7.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[10:31] * DLange (~DLange@dlange.user.oftc.net) Quit (Quit: gaaaaaaahhhhhhhhh)
[10:33] * DLange (~DLange@dlange.user.oftc.net) has joined #ceph
[10:36] <niklas> hi
[10:36] <niklas> my cluster reports:
[10:36] <niklas> HEALTH_WARN 11 pgs backfill; 8 pgs backfill_toofull; 14 pgs backfilling; 14 pgs degraded; 25 pgs stuck unclean; recovery 74227/25801329 degraded (0.288%); 1 near full osd(s)
[10:37] <niklas> Why won't it continue repairing?
[10:37] * guppy_ (~quassel@guppy.xxx) has joined #ceph
[10:37] * bandrus (~Adium@cpe-76-95-217-129.socal.res.rr.com) Quit (Quit: Leaving.)
[10:37] <niklas> there is one osd which is 86% full (as of "df")
[10:39] * shang (~ShangWu@122-116-16-162.HINET-IP.hinet.net) has joined #ceph
[10:40] * sleinen (~Adium@2001:620:0:25:4864:b9d2:87b5:bee7) has joined #ceph
[10:40] * guppy (~quassel@guppy.xxx) Quit (Ping timeout: 480 seconds)
[10:48] <andreask> niklas: because of this near full osds the backfill is rejected ... you could increase the "backfill full ratio" or better add a more OSDs
[10:49] <niklas> andreask: ok
[10:49] * Midnightmyth (~quassel@93-167-84-102-static.dk.customer.tdc.net) has joined #ceph
[10:49] <niklas> andreask: but why are the pgs balanced that badly?
[10:50] <niklas> ceph -s tells me:
[10:50] <niklas> 117 TB data, 234 TB used, 95082 GB / 327 TB avail;
[10:50] <mattt> how does ceph's radosgw perform in comparison to openstack swift?
[10:52] * yanzheng (~zhyan@ has joined #ceph
[10:53] <niklas> All my osds are equal, but on one OSD I have 67 PGs, and on another 125 !
[10:54] <andreask> niklas: depends on the number of pgs you choose for a pool how evenly they are distributed ... and also your chrush map has influence
[10:54] <niklas> I have 2 Hosts, 90 OSDs (all OSDs equal) and 4k PGs
[10:54] <mattt> 90 osds on 2 hosts? wow :)
[10:55] <niklas> With a replica Size of 2 I get just about 100 PGs per OSD - or at least I should get
[10:57] <andreask> and your pools have all a fairly tuned number of pgs?
[11:01] <niklas> I have the three "default" pools with 64 PGs each and one main pool with 4k PGs
[11:01] <niklas> (its a test cluster)
[11:02] * claenjoy (~leggenda@ has joined #ceph
[11:02] <niklas> andreask: Recovering works now, but I still don't understand why the PGs are distributed that badly…
[11:07] <pieter> is it ok to run mds and osd on the same host, or should I rather move the mds to a monitor?
[11:07] <pieter> using cephfs
[11:07] * jks (~jks@3e6b5724.rev.stofanet.dk) has joined #ceph
[11:09] * bandrus (~Adium@cpe-76-95-217-129.socal.res.rr.com) has joined #ceph
[11:11] * hybrid512 (~walid@LPoitiers-156-86-25-85.w193-248.abo.wanadoo.fr) has joined #ceph
[11:11] * hybrid5121 (~walid@LPoitiers-156-86-25-85.w193-248.abo.wanadoo.fr) Quit (Read error: Connection reset by peer)
[11:15] <andreask> niklas: have you tried setting some crush tunables?
[11:15] <niklas> nope
[11:15] <niklas> just the default crushmap
[11:17] <andreask> niklas: might be worth trying, especially as it is a test cluster ... http://ceph.com/docs/master/rados/operations/crush-map/#tuning-crush-the-hard-way
[11:18] <niklas> what would I want to tune?
[11:24] * guppy (~quassel@guppy.xxx) has joined #ceph
[11:25] * guppy_ (~quassel@guppy.xxx) Quit (Read error: Connection reset by peer)
[11:26] <claenjoy> hi everyone , with a ceph-cluster with 4 osds , how many "ceph osd pool create volumes and images" I can create ?
[11:27] * thomnico (~thomnico@ Quit (Quit: Ex-Chat)
[11:27] <andreask> niklas: I'd give that example there a try
[11:28] <claenjoy> the cluster-ceph is big 800gb
[11:30] * jcsp (~john@82-71-55-202.dsl.in-addr.zen.co.uk) Quit (Ping timeout: 480 seconds)
[11:30] <niklas> andreask: will that have an effekt on pre existing pools?
[11:31] <andreask> niklas: I don't think so
[11:31] <niklas> ok, thats a pity
[11:32] <andreask> niklas: do many of your osds have that difference in number of pgs?
[11:34] <niklas> andreask: I wrote myself a small script that parses pgdump:
[11:34] <niklas> http://pastebin.com/DR5NQHi0
[11:34] <niklas> If it was distributed perfectly, every osd would have 97 PGs
[11:36] <andreask> hmm ... well, but I don't think that looks that bad
[11:36] * sleinen (~Adium@2001:620:0:25:4864:b9d2:87b5:bee7) Quit (Quit: Leaving.)
[11:36] <pieter> can anyone please point me in the right direction to troubleshoot:
[11:36] * sleinen (~Adium@ has joined #ceph
[11:37] <pieter> root@cephfs2:~# ceph health
[11:37] <pieter> HEALTH_WARN 58 pgs backfill; 10 pgs backfilling; 68 pgs stuck unclean; recovery 1646137/7929297 degraded (20.760%); recovering 69 o/s, 57129B/s
[11:37] * bandrus (~Adium@cpe-76-95-217-129.socal.res.rr.com) Quit (Quit: Leaving.)
[11:37] <pieter> just added another OSD (new disk)
[11:37] <pieter> but that seems VERY slow for backfill?
[11:38] <niklas> andreask: The Problem is, that with one OSD having 125 PGs (OSD 43), when that OSD is full, all the cluster is full even though it is only filled up to like 75% of its total size
[11:42] * rendar (~s@host228-182-dynamic.20-87-r.retail.telecomitalia.it) has joined #ceph
[11:43] * jcsp (~john@82-71-55-202.dsl.in-addr.zen.co.uk) has joined #ceph
[11:44] * sleinen (~Adium@ Quit (Ping timeout: 480 seconds)
[11:48] <andreask> niklas: hmm ... beside increasing that limit for the nearly full value you could try to deactivate that osd and wait for the pgs to be redistributed
[11:49] <niklas> andreask: Its a testcluster anyways, so not that important. I just need to be sure that in production the PGs will be distributed more or less evenly so I can be sure of how much data I can fit onto the cluster…
[11:50] <niklas> andreask: I can't afford to loose 25% of my storage space because the PGs won't be distributed evenly…
[11:51] <niklas> (Lunchtime now, I'll be back in about one hour)
[11:52] <mattt> can you decrease the # of pgs on a pool ?
[11:55] <mattt> appears not
[12:01] * ScOut3R (~ScOut3R@catv-89-133-25-52.catv.broadband.hu) Quit (Remote host closed the connection)
[12:02] <andreask> mattt: you can create a new pool, use "rados cppool" to transfer the data and then rename/remove the old pool and then rename the new pool
[12:02] <mattt> andreask: yep, doing that now, found it on sebastian's blog :P
[12:03] * sleinen (~Adium@user-23-16.vpn.switch.ch) has joined #ceph
[12:15] * andreask (~andreask@h081217135028.dyn.cm.kabsi.at) Quit (Ping timeout: 480 seconds)
[12:17] * sjm (~sjm@dhcp-108-168-18-236.cable.user.start.ca) has joined #ceph
[12:18] * torment (~torment@pool-96-228-149-152.tampfl.fios.verizon.net) Quit (Ping timeout: 480 seconds)
[12:18] * torment (~torment@pool-96-228-149-152.tampfl.fios.verizon.net) has joined #ceph
[12:26] * ismell (~ismell@host-64-17-89-79.beyondbb.com) has joined #ceph
[12:28] * ismell_ (~ismell@host-64-17-89-79.beyondbb.com) Quit (Ping timeout: 480 seconds)
[12:38] * guppy_ (~quassel@guppy.xxx) has joined #ceph
[12:40] * guppy (~quassel@guppy.xxx) Quit (Read error: Connection reset by peer)
[12:54] * sleinen (~Adium@user-23-16.vpn.switch.ch) Quit (Quit: Leaving.)
[12:57] <jtang> hey all
[12:57] <rsanti> hi, is there a way to diagnose rbd? when we launch it just hangs, waiting on some socket that never returns any data
[12:57] <jtang> we're trying to write up short evaluation report, and ceph is included in it
[12:57] <rsanti> (for those who were following yesterday night, ceph now responds to commands)
[12:58] <jtang> we're trying to find out what the max filesystem size is for ceph
[12:58] <jtang> does anyone know where i can find that out? of if its known?
[13:05] * peetaur (~peter@CPEbc1401e60493-CMbc1401e60490.cpe.net.cable.rogers.com) has joined #ceph
[13:11] * andreask (~andreask@h081217135028.dyn.cm.kabsi.at) has joined #ceph
[13:11] * ChanServ sets mode +v andreask
[13:13] * sleinen (~Adium@ext-dhcp-174.eduroam.unibe.ch) has joined #ceph
[13:13] <niklas> andreask: I just deleted all pools and create a new one with 4500 PGs. I ended up with one osd with 75 PGs and one with 123 PGs
[13:13] <WebSpider> hi all! has anyone ever tried to run ceph as a tor hidden service and survived? :)
[13:13] <niklas> (I have 90 OSDs, with a repilca Size of 2 every OSD should have about 100 PGs)
[13:14] <niklas> andreask: that was after tuning the crushmap
[13:17] <andreask> niklas: and all osd disks are of the same size and you have exactly half of them per host?
[13:18] <andreask> ...so all symmetric
[13:18] <niklas> Yep
[13:18] * sleinen1 (~Adium@2001:620:0:26:6994:fd14:c1c8:90d6) has joined #ceph
[13:18] <niklas> 4T Disks, 45 each Host
[13:21] * sleinen (~Adium@ext-dhcp-174.eduroam.unibe.ch) Quit (Ping timeout: 480 seconds)
[13:23] <andreask> niklas: sorry, no idea if there is another tuning possible here to prevent such differences ... maybe a developer on the mailing list ore later here in the channel can help
[13:24] <niklas> andreask: I just noticed that its always the same osd which got most of the pgs
[13:24] <jtang> heh, sounds you got some backblaze pods?
[13:24] <niklas> Also when I have two pools with each 4500 PGs, each osd will have doubled the amount of pgs
[13:25] <andreask> makes sense
[13:25] <niklas> Is that jenkins algorithm dterministic?
[13:26] <andreask> niklas: would be good to find the data :-)
[13:26] <andreask> so ... definitely deteministic
[13:26] <niklas> yeah, but I meant you could put some seed into it…
[13:26] <niklas> like the name of the pool or something
[13:26] <niklas> but yeah, that makes sense…
[13:27] <niklas> andreask: thank you very much, I think I'll ask at ceph-users
[13:27] <andreask> yw
[13:28] * sleinen1 (~Adium@2001:620:0:26:6994:fd14:c1c8:90d6) Quit (Remote host closed the connection)
[13:30] * ScOut3R (~ScOut3R@254C2A51.mobile.pool.telekom.hu) has joined #ceph
[13:31] * KevinPerks (~Adium@cpe-066-026-252-218.triad.res.rr.com) has joined #ceph
[13:31] * ScOut3R (~ScOut3R@254C2A51.mobile.pool.telekom.hu) Quit (Remote host closed the connection)
[13:35] * sleinen1 (~Adium@2001:620:0:25:c1c5:6e17:8a38:4f8c) has joined #ceph
[13:36] * bandrus (~Adium@cpe-76-95-217-129.socal.res.rr.com) has joined #ceph
[13:38] * wschulze (~wschulze@cpe-72-229-37-201.nyc.res.rr.com) has joined #ceph
[13:40] * sjm (~sjm@dhcp-108-168-18-236.cable.user.start.ca) Quit (Read error: Operation timed out)
[13:55] * yanzheng (~zhyan@ Quit (Remote host closed the connection)
[13:56] * sjm (~sjm@dhcp-108-168-18-236.cable.user.start.ca) has joined #ceph
[14:09] * alfredodeza (~alfredode@c-24-131-46-23.hsd1.ga.comcast.net) has joined #ceph
[14:11] * kraken (~kraken@c-24-131-46-23.hsd1.ga.comcast.net) has joined #ceph
[14:11] * odyssey4me2 (~odyssey4m@ has joined #ceph
[14:12] * yanzheng (~zhyan@ has joined #ceph
[14:13] <odyssey4me2> jtang - for cephfs or for an image/pool/cluster?
[14:13] * odyssey4me (~odyssey4m@ Quit (Ping timeout: 480 seconds)
[14:13] * odyssey4me2 (~odyssey4m@ Quit ()
[14:14] * odyssey4me (~odyssey4m@ has joined #ceph
[14:23] * pieter (~pieter@105-236-215-222.access.mtnbusiness.co.za) Quit (Quit: Konversation terminated!)
[14:31] * nhm (~nhm@ Quit (Quit: Lost terminal)
[14:32] * jcfischer (~fischer@user-23-16.vpn.switch.ch) has joined #ceph
[14:38] * ScOut3R (~scout3r@4E5C2305.dsl.pool.telekom.hu) has joined #ceph
[14:43] * kyann (~oftc-webi@tui75-3-88-168-236-26.fbx.proxad.net) has joined #ceph
[14:47] * thomnico (~thomnico@2a01:e35:8b41:120:40bd:43f8:c1b6:5f00) has joined #ceph
[14:49] <claenjoy> question about openstack and ceph , I have 3 nodes 1 controller and 2 computes / controller run cinder-volume and glace-api / and the 2 compute nodes runs nova-compute / from the here : http://ceph.com/docs/master/rbd/rbd-openstack/ | Do I need to install cinder and glance in the 2 compute nodes ?
[14:50] <claenjoy> andreask: what do you think about my question ?
[14:53] * sleinen1 (~Adium@2001:620:0:25:c1c5:6e17:8a38:4f8c) Quit (Ping timeout: 480 seconds)
[14:54] <claenjoy> my confusion, it is I have 2 compute nodes so I have 2 uuid of the secret , then in configuration of cinder.conf need to add rbd_secret_uuid={uuid of secret} but I have 2 uuid so I'm not sure if I did the right step
[14:58] * kyann (~oftc-webi@tui75-3-88-168-236-26.fbx.proxad.net) Quit (Quit: Page closed)
[15:01] <andreask> claenjoy: you generate one uuid, define that on the compute node in the cinder configuration and use this uuid on all your compute nodes to define the secret
[15:03] <jtang> odyssey4me: cephfs, pools, rbd
[15:03] <claenjoy> andreask: aaa ok !!! thanks a lot ! I generated 2 :/ , I should use the last one , correct ?
[15:03] <andreask> doesn't matter
[15:03] <jtang> we're interested in knowing what the architectural limits are in terms of capacity
[15:03] <claenjoy> andreask: mmm
[15:04] <jtang> it isnt't clear to me where that information is stored onthe ceph documentation site
[15:04] <andreask> claenjoy: you just need to decide for one
[15:05] * markbby (~Adium@ has joined #ceph
[15:05] <claenjoy> andreask: but I need to install cinder-volume and glace-api in the 2 compute nodes? I have multi-node
[15:10] * yanzheng (~zhyan@ Quit (Remote host closed the connection)
[15:17] * sleinen (~Adium@ext-dhcp-174.eduroam.unibe.ch) has joined #ceph
[15:18] <JustEra> alfredodeza, Hey, just upgraded the ceph-deploy with the latest git source and it works now don't know why it didn't worked last night
[15:20] * bandrus (~Adium@cpe-76-95-217-129.socal.res.rr.com) Quit (Quit: Leaving.)
[15:20] * markbby1 (~Adium@ has joined #ceph
[15:20] * markbby (~Adium@ Quit (Remote host closed the connection)
[15:20] <andreask> claenjoy: no need for cinder-volume on the compute nodes, you can install glance-api if you want local image caching
[15:21] * diegows (~diegows@ has joined #ceph
[15:22] * yanzheng (~zhyan@ has joined #ceph
[15:22] <alfredodeza> JustEra: :)
[15:23] <claenjoy> andreask: perfect, thanks ! I m trying to use the same uuid in the other compute node ...
[15:24] * julian (~julianwa@ has joined #ceph
[15:24] * sagelap (~sage@ Quit (Ping timeout: 480 seconds)
[15:25] * sleinen (~Adium@ext-dhcp-174.eduroam.unibe.ch) Quit (Ping timeout: 480 seconds)
[15:25] * sleinen (~Adium@2001:620:0:25:9940:9e10:f6a6:c38b) has joined #ceph
[15:31] * jeff-YF (~jeffyf@ has joined #ceph
[15:32] <claenjoy> andreask: DONe ! some secret for both nodes !
[15:38] * sagelap (~sage@ has joined #ceph
[15:45] * dmsimard (~Adium@ap02.wireless.co.mtl.iweb.com) has joined #ceph
[15:52] * sagelap (~sage@ Quit (Ping timeout: 480 seconds)
[15:52] * med is now known as peg_leg_med
[16:01] * jcfischer (~fischer@user-23-16.vpn.switch.ch) Quit (Quit: jcfischer)
[16:12] * PerlStalker (~PerlStalk@2620:d3:8000:192::70) has joined #ceph
[16:13] * sagelap (~sage@ has joined #ceph
[16:17] * xmltok (~xmltok@cpe-76-170-26-114.socal.res.rr.com) has joined #ceph
[16:19] * xmltok (~xmltok@cpe-76-170-26-114.socal.res.rr.com) Quit ()
[16:23] * xmltok (~xmltok@cpe-76-170-26-114.socal.res.rr.com) has joined #ceph
[16:26] * b1tbkt (~b1tbkt@24-217-192-155.dhcp.stls.mo.charter.com) has joined #ceph
[16:30] <odyssey4me> jtang - cephfs' limit is based on the mds config, which by default is 1TB. This can be adjusted with the 'mds max file size' conf entry.
[16:32] <odyssey4me> jtang - pools, etc are limited by whatever's using them as far as I know... rather than ceph itself... ceph doesn't care about the size of a pool/cluster... it'll grow to petabytes and beyond
[16:33] <odyssey4me> ie if you're using an image as a block mount for a server, the image could be very, very large... but the file system used on the image by the server mounting it would limit its size
[16:36] * jeff-YF (~jeffyf@ Quit (Quit: jeff-YF)
[16:40] * julian (~julianwa@ Quit (Quit: afk)
[16:42] * jeff-YF (~jeffyf@ has joined #ceph
[16:42] <jtang> odyssey4me: i wasnt refering to the max filesize
[16:43] <jtang> we're writing some docs to "sell" it to managers, so having a theoretical or tested number would be nice to present
[16:44] <odyssey4me> jtang - then you're going to have to looking for the various papers/performance tests that have been done, or ask an Inktank rep
[16:44] <odyssey4me> there are some that hang around here that can share their numbers
[16:44] <odyssey4me> but you're going to have to be more specific about what you want
[16:45] <odyssey4me> are you looking for how big a cluster can grow in total space?
[16:45] <odyssey4me> or how many osd's it can handle?
[16:46] <odyssey4me> or samples of what people have in terms of large pools/images?
[16:48] * jeff-YF (~jeffyf@ Quit (Quit: jeff-YF)
[16:50] * nwat (~nwat@c-50-131-197-174.hsd1.ca.comcast.net) has joined #ceph
[16:54] * jeff-YF (~jeffyf@ has joined #ceph
[16:55] * JustEra (~JustEra@ Quit (Quit: This computer has gone to sleep)
[16:56] * angdraug (~angdraug@c-98-248-39-148.hsd1.ca.comcast.net) has joined #ceph
[16:57] * dmsimard1 (~Adium@ has joined #ceph
[16:57] <jtang> odyssey4me: im gonna have a look around to see if i can find out more
[16:58] * BillK (~BillK-OFT@124-169-207-19.dyn.iinet.net.au) Quit (Ping timeout: 480 seconds)
[16:58] <jtang> just throwing around "ceph will scale to petabytes of storage" and refering to the website is abit of a fluffy answer
[16:59] <jtang> and saying im just limited by hardware isnt a great answer :P
[16:59] <jtang> but thanks for the ideas on where to look
[17:00] * sleinen (~Adium@2001:620:0:25:9940:9e10:f6a6:c38b) Quit (Ping timeout: 480 seconds)
[17:02] * dmsimard (~Adium@ap02.wireless.co.mtl.iweb.com) Quit (Read error: Connection reset by peer)
[17:04] * Vjarjadian (~IceChat77@ has joined #ceph
[17:08] * twcai (~twcai@ has joined #ceph
[17:16] <swinchen> Any idea what might cause this? http://pastie.org/pastes/8339146/text
[17:16] * DarkAceZ (~BillyMays@ has joined #ceph
[17:17] <alfredodeza> swinchen: what version of ceph-deploy are you using
[17:18] <swinchen> alfredodeza: 1.2.6
[17:20] <alfredodeza> swinchen: yep, I know what is going on here
[17:20] <alfredodeza> hrmn
[17:21] * yanzheng (~zhyan@ Quit (Remote host closed the connection)
[17:21] <alfredodeza> the subprocess command to find where `lsb_release` lives cannot find `which` because the $PATH lacks where that is located
[17:21] <alfredodeza> just to confirm
[17:21] <alfredodeza> can you add the PATH to /etc/environment ?
[17:22] <swinchen> alfredodeza: "yum install which" fixed the problem. I am shocked that fedora doesn't install it by default. At least not the cloud image.
[17:22] <kraken> http://i.imgur.com/Q4bI5.gif
[17:22] <alfredodeza> swinchen: this is terrible :(
[17:22] * allsystemsarego (~allsystem@ Quit (Quit: Leaving)
[17:22] <swinchen> alfredodeza: How did you know it couldn't find "which"? I don't see that anywhere in the output of the debug information.
[17:23] <alfredodeza> swinchen: that is part of why this is so terrible :(
[17:23] <alfredodeza> I've lost my mind to similar bugs before
[17:23] <alfredodeza> that is how I know
[17:23] <swinchen> Haha, thankfully the community has you around!
[17:25] <alfredodeza> swinchen: I've created issue 6353
[17:25] <kraken> alfredodeza might be talking about: http://tracker.ceph.com/issues/6353 [Fedora installs fail because `which` is not installed by default]
[17:25] <alfredodeza> this is not a difficult fix for ceph-deploy, should get out on the next release
[17:27] <swinchen> thanks alfredodeza!
[17:28] <swinchen> Now I am hitting this problem... ceph installs fine, bit when I try to create a monitor I get a number of errors. http://pastie.org/pastes/8339172/text
[17:29] * sjustlaptop (~sam@24-205-35-233.dhcp.gldl.ca.charter.com) has joined #ceph
[17:29] <swinchen> I disabled selinux (setenforce 0)
[17:30] * odyssey4me (~odyssey4m@ Quit (Ping timeout: 480 seconds)
[17:33] <alfredodeza> swinchen: why root doesn't have permissions? this line is odd: bash: line 0: ulimit: open files: cannot modify limit: Operation not permitted
[17:33] * BManojlovic (~steki@ Quit (Quit: Ja odoh a vi sta 'ocete...)
[17:33] <swinchen> alfredodeza: I know. I don't understand it.
[17:33] * yanzheng (~zhyan@ has joined #ceph
[17:33] <alfredodeza> oh PAM... that could be it
[17:34] <alfredodeza> in `cat /etc/ssh/sshd_config` do you have `UsePAM no` ?
[17:35] <swinchen> alfredodeza: UsePAM yes
[17:35] <alfredodeza> hrmn
[17:36] <swinchen> I found this bug, but it doesn't seem to be about the ulimit error: http://tracker.ceph.com/issues/4036
[17:37] * xiaoxi (~xiaoxi@ Quit (Remote host closed the connection)
[17:39] * andreask (~andreask@h081217135028.dyn.cm.kabsi.at) Quit (Ping timeout: 480 seconds)
[17:39] <claenjoy> andreask: I have 4 osds , 3 of them are tradition hds and 1 is only ssd , in this case IS better activate the journal only in the ssd hd ?
[17:40] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[17:40] <alfredodeza> swinchen: what about `cat /etc/security/limits.conf`
[17:40] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[17:41] <alfredodeza> just found that some of our internal test tools modify that
[17:42] <swinchen> alfredodeza: http://pastie.org/8339199 (seems to be empty once you remove comments)
[17:42] * kleini (~marcus@2001:4dd0:ff00:88c0:f66d:4ff:fe15:b1c3) Quit (Quit: Konversation terminated!)
[17:44] <swinchen> alfredodeza: this is only file in the limits.d directory: http://pastie.org/pastes/8339202/text
[17:44] <alfredodeza> swinchen: try adding this to *that* file:
[17:44] <alfredodeza> root soft nofile unlimited
[17:47] <swinchen> alfredodeza: no good. Same problem. I also tried replacing "root" with "ceph"
[17:47] <alfredodeza> hrmn
[17:49] <swinchen> I would try Ubuntu... but... I just hate ubuntu :/
[17:50] <alfredodeza> sorry, I am looking into old threads
[17:50] <alfredodeza> but I can't tell what is the fix here
[17:50] <swinchen> Thats alright, thanks alfredodeza
[17:51] <alfredodeza> I am actually trying to find a link so I can share
[17:51] <alfredodeza> but I can't find that
[17:51] <xdeller> joao: hi, what you can say if ceph heap stats jumping in VSS between 600M and 10G for MON? real stat keeps as 12G VSS and 370M resident memory
[17:51] <alfredodeza> let me get you a paste of that thread as I find that maybe something can help you there
[17:51] <alfredodeza> that talks exactly about this issue
[17:52] <alfredodeza> swinchen: http://fpaste.org/40750/60593113/
[17:52] <alfredodeza> try that before going to Ubuntu :)
[17:54] <mattt> doh, missed early bird tickets for ceph day london :(
[17:55] <infernix> [ERROR ] IOError: [Errno 2] No such file or directory: '/var/lib/ceph/tmp/ceph-disk.prepare.lock'
[17:56] <swinchen> alfredodeza: I added "* soft core unlimited" to limits.conf without success. hrm
[17:56] <swinchen> Do I need to restart something?
[17:57] <alfredodeza> not sure :/
[17:57] <swinchen> heck, I will just reboot
[17:57] <swinchen> that will restart something :P
[17:58] <infernix> i think ceph-deploy purgedata is a bit too ambitious
[17:58] <infernix> it should clean out data, not rm -rf /var/lib/ceph - that breaks stuff
[17:59] <swinchen> alfredodeza: It got closer after a reboot! http://pastie.org/pastes/8339245/text
[17:59] * yanzheng (~zhyan@ Quit (Ping timeout: 480 seconds)
[17:59] <alfredodeza> nice
[18:00] <swinchen> It looks like it did start the monitor despite the error.
[18:00] <swinchen> alfredodeza: Thanks for the help!
[18:00] <alfredodeza> no problem
[18:00] <alfredodeza> that permission denied though
[18:00] <alfredodeza> weird
[18:00] <claenjoy> I m trying to install ubuntu-12.04-desktop in volume but after start the installation is stop and the output of ceph -w : said 0 op/s
[18:01] <swinchen> I am following the guide exactly. I am not sure what I could be doing wrong.
[18:01] <alfredodeza> swinchen: can you try running that command directly and see if you get any output? --> ceph daemon mon.ceph-test mon_status
[18:01] <swinchen> alfredodeza: http://pastie.org/8339257
[18:02] <infernix> [ERROR ] INFO:ceph-disk:Will colocate journal with data on /dev/sdb
[18:02] <alfredodeza> hrmn, maybe we need to wait a bit before calling mon_status
[18:02] <infernix> is that really an error? o_O
[18:02] <alfredodeza> infernix: ceph-deploy will log to ERROR for all stderr output in the remote host
[18:03] <alfredodeza> and DEBUG for stdout
[18:05] * grepory (~Adium@50-115-70-146.static-ip.telepacific.net) has joined #ceph
[18:07] * Kioob`Taff (~plug-oliv@local.plusdinfo.com) Quit (Quit: Leaving.)
[18:07] * mattt (~mattt@ Quit (Read error: Connection reset by peer)
[18:07] * nwat (~nwat@c-50-131-197-174.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[18:08] <joao> xdeller, paste the output from heap stats please?
[18:12] <xdeller> joao, here please http://pastie.org/8339283
[18:14] <joao> xdeller, is there anything interesting happening for that particular monitor?
[18:15] <joao> out of the quorum, synchronizing, anything?
[18:15] <joao> also, what version?
[18:15] <xdeller> one i had restarted last night, nothing special except leader property
[18:15] <xdeller> 0.61.8
[18:16] <joao> in this case it might just be that tcmalloc is not relinquishing pages to the OS
[18:16] <joao> try calling a heap release and recheck stats
[18:16] <xdeller> I had replied in list, it currently have twelve gigs of virtual commit, this may be the reason for incorrect stat reporting
[18:16] <xdeller> did some minutes ago with no difference before/after
[18:18] <joao> oh, wait
[18:18] <joao> are you targeting your commands to a single monitor?
[18:18] <joao> assuming you have more than one monitor
[18:18] <xdeller> yes
[18:18] <joao> okay
[18:18] <joao> then try 'heap dump'
[18:18] <joao> and send us the file
[18:18] * odyssey4me (~odyssey4m@41-133-58-101.dsl.mweb.co.za) has joined #ceph
[18:19] <joao> or drop it on cephdrop
[18:19] <joao> oh, no you're not
[18:19] <joao> you're not targeting just one monitor
[18:19] <joao> all your ceph heap stats are being sent to different monitors
[18:19] <joao> hence the memory usage differences
[18:20] <xdeller> oh
[18:20] <xdeller> I see
[18:20] <joao> xdeller, try running with -m ip:port
[18:20] <xdeller> there is a tip in the tcmalloc heap stats:
[18:21] * a (~a@ has joined #ceph
[18:21] <joao> ReleaseFreeMemory()?
[18:21] <joao> that's what 'release' does :)
[18:21] <xdeller> mon.0tcmalloc heap stats:-- mon.3tcmalloc heap stats:-- etc
[18:21] <xdeller> i missed it
[18:21] <joao> ah yes
[18:21] <joao> that
[18:21] * a is now known as Guest7188
[18:23] <xdeller> so can you suggest anything to catch on restart of next monitor when same memory peak will happen?
[18:24] * odyssey4me2 (~odyssey4m@ has joined #ceph
[18:25] * xmltok_ (~xmltok@pool101.bizrate.com) has joined #ceph
[18:26] * KindTwo (~KindOne@h89.16.131.174.dynamic.ip.windstream.net) has joined #ceph
[18:26] * odyssey4me (~odyssey4m@41-133-58-101.dsl.mweb.co.za) Quit (Read error: Operation timed out)
[18:28] * KindOne (~KindOne@0001a7db.user.oftc.net) Quit (Ping timeout: 480 seconds)
[18:28] * KindTwo is now known as KindOne
[18:31] * nwat (~nwat@eduroam-237-79.ucsc.edu) has joined #ceph
[18:32] * peg_leg_med is now known as med_caught_a_cephalopod
[18:33] * gaveen (~gaveen@ has joined #ceph
[18:37] * Tamil1 (~Adium@cpe-108-184-71-119.socal.res.rr.com) has joined #ceph
[18:37] * sprachgenerator (~sprachgen@ has joined #ceph
[18:38] * med_caught_a_cephalopod is now known as pegleg-med
[18:39] * bandrus (~Adium@cpe-76-95-217-129.socal.res.rr.com) has joined #ceph
[18:41] * claenjoy (~leggenda@ Quit (Quit: Leaving.)
[18:42] * claenjoy (~leggenda@ has joined #ceph
[18:42] * claenjoy (~leggenda@ Quit ()
[18:44] * claenjoy (~leggenda@ has joined #ceph
[18:44] * claenjoy (~leggenda@ Quit ()
[18:44] * claenjoy (~leggenda@ has joined #ceph
[18:45] * claenjoy (~leggenda@ Quit ()
[18:45] * ircolle (~Adium@c-67-165-237-235.hsd1.co.comcast.net) has joined #ceph
[18:46] * pegleg-med is now known as pepebarba
[18:47] * odyssey4me (~odyssey4m@41-133-58-101.dsl.mweb.co.za) has joined #ceph
[18:48] <joao> xdeller, does this always happen on start?
[18:49] * claenjoy (~leggenda@ has joined #ceph
[18:49] * claenjoy (~leggenda@ Quit ()
[18:50] <xdeller> will check today at night
[18:50] <xdeller> it is production env
[18:51] * claenjoy (~leggenda@ has joined #ceph
[18:51] * rudolfsteiner (~federicon@ has joined #ceph
[18:51] * Cube (~Cube@66-87-64-12.pools.spcsdns.net) Quit (Quit: Leaving.)
[18:52] * claenjoy (~leggenda@ Quit ()
[18:52] * odyssey4me2 (~odyssey4m@ Quit (Ping timeout: 480 seconds)
[18:53] * claenjoy (~leggenda@ has joined #ceph
[18:55] * pepebarba is now known as med
[18:55] * odyssey4me (~odyssey4m@41-133-58-101.dsl.mweb.co.za) Quit (Ping timeout: 480 seconds)
[18:58] * Vjarjadian (~IceChat77@ Quit (Quit: Make it idiot proof and someone will make a better idiot.)
[19:02] * sjusthm (~sam@24-205-35-233.dhcp.gldl.ca.charter.com) has joined #ceph
[19:06] * dpippenger (~riven@cpe-76-166-208-83.socal.res.rr.com) Quit (Quit: Leaving.)
[19:08] * bandrus (~Adium@cpe-76-95-217-129.socal.res.rr.com) Quit (Quit: Leaving.)
[19:08] * xarses (~andreww@ has joined #ceph
[19:08] * bandrus (~Adium@cpe-76-95-217-129.socal.res.rr.com) has joined #ceph
[19:13] * davidzlap (~Adium@cpe-75-84-249-188.socal.res.rr.com) has joined #ceph
[19:16] <joao> xdeller, kay
[19:16] * thomnico (~thomnico@2a01:e35:8b41:120:40bd:43f8:c1b6:5f00) Quit (Quit: Ex-Chat)
[19:17] * ishkabob (~c7a82cc0@webuser.thegrebs.com) has joined #ceph
[19:20] <ishkabob> hey guys, does anyone have some starting points for troubleshooting slow requests in ceph
[19:21] <ishkabob> specifically, we are seeing slow requests when we reboot one of the switches in an active/active switch setup, which shouldn't happe
[19:35] <sjusthm> ishkabob: first guess would be that the osds are losing connectivity
[19:38] <swinchen> so... "ceph-deploy osd destroy" is not implemented... where in the documentation can I find out how to do that manually?
[19:39] * gregmark (~Adium@ has joined #ceph
[19:39] * sjm (~sjm@dhcp-108-168-18-236.cable.user.start.ca) Quit (Ping timeout: 480 seconds)
[19:39] <mech4221> anyone using mako templates ?
[19:39] <mech4221> oops - wrong channel - sorry
[19:49] * sjm (~sjm@dhcp-108-168-18-236.cable.user.start.ca) has joined #ceph
[19:56] <ishkabob> sjusthm
[19:56] <sjusthm> ?
[19:56] <ishkabob> sjusthm: yeah i figure that as well, anything else i should be looking for in the logs?
[19:57] <ishkabob> each node is configured with a 40gigabit bonded ethernet (10Gx4)
[19:57] <sjusthm> ishkabob: well, I would troubleshoot the network
[19:57] <ishkabob> i would imagine that's enough bandwidth :)
[19:57] <ishkabob> yeah, i'm not seeing any significant drops on the interfaces
[19:57] <ishkabob> and the clients don't seem to be dropping their RBD mounts or anything
[20:01] <swinchen> So this doesn't look quite right: http://pastie.org/pastes/8339551/text
[20:01] <ishkabob> sjusthm: any chance modifying the modifying osd op thread timeout and osd op complaint time would have an effect?
[20:03] <sjusthm> ishkabob: if it happens when you reboot a switch, then I would bet that the op thread won't matter
[20:03] <sjusthm> osd op complaint time would just make the complaint go away
[20:04] * xmltok_ (~xmltok@pool101.bizrate.com) Quit (Quit: Bye!)
[20:05] * tziOm (~bjornar@ti0099a340-dhcp0395.bb.online.no) has joined #ceph
[20:05] <sjusthm> ishkabob: you could reproduce with debug ms = 5, debug osd = 5 on the osds
[20:05] <ishkabob> sjusthm: How does one limit the amount of time a slow request can exist before it is retried
[20:05] <sjusthm> doesn't work that way, it's slow because the osd can't process it yet
[20:05] <sjusthm> perhaps because it can't contact a replica
[20:06] <ishkabob> sjusthm: how do I set those debug levels? can they be set on the live system?
[20:07] <sjusthm> yes
[20:07] <sjusthm> injectargs
[20:07] <sjusthm> what version are you running?
[20:08] <ishkabob> ceph version 0.67.2
[20:08] <sjusthm> ceph tell osd.* injectargs '--debug_osd 5 --debug_ms 5'
[20:08] <sjusthm> upi
[20:09] <sjusthm> **you'll want to escape the *
[20:09] * xmltok_ (~xmltok@pool101.bizrate.com) has joined #ceph
[20:11] <swinchen> How can I do about trouble shooting this: health HEALTH_WARN 174 pgs degraded; 192 pgs stuck unclean
[20:11] * Xiol_ (~Xiol@94-193-254-111.zone7.bethere.co.uk) has joined #ceph
[20:11] <ishkabob> swinchen: that should be a recoverable problem, is the number of pgs degraded going down?
[20:11] * xmltok_ (~xmltok@pool101.bizrate.com) Quit ()
[20:12] * xmltok_ (~xmltok@pool101.bizrate.com) has joined #ceph
[20:12] * dpippenger (~riven@tenant.pas.idealab.com) has joined #ceph
[20:12] <swinchen> ishkabob: No. I don't even have any data stored on the cluster yet.
[20:12] <ishkabob> are all of your osds up?
[20:12] <ishkabob> you can tell with "ceph osd tree"
[20:13] <swinchen> ishkabob: they appear to be, yes.
[20:14] <ishkabob> try restarting all the osds and mons
[20:15] * ircolle (~Adium@c-67-165-237-235.hsd1.co.comcast.net) Quit (Quit: Leaving.)
[20:15] * alfredodeza is now known as alfredo|noms
[20:15] <swinchen> ishkabob: no change.
[20:15] <swinchen> :/
[20:15] <ishkabob> how many nodes are you running?
[20:16] <swinchen> ishkabob: one right now (virtual machine) w/ 2 disks 2 OSDs
[20:16] <gregaf> ishkabob: swinchen: probably crush tunables; check the docs
[20:16] <gregaf> </driveby>
[20:16] <swinchen> I did set this when readding the guide: osd crush chooseleaf type = 0
[20:17] <ishkabob> swinchen: yeah gregaf is probably right, you have to do some crush tuning when you're only running 1 node
[20:17] * Xiol (~Xiol@94-193-254-111.zone7.bethere.co.uk) Quit (Ping timeout: 480 seconds)
[20:17] <ishkabob> you could always just spin yourself up another node :)
[20:17] <swinchen> Instead of doing that I think I will start a second node :)
[20:17] <ishkabob> boom
[20:17] <swinchen> Thanks, I will give that a try.
[20:22] * nwat (~nwat@eduroam-237-79.ucsc.edu) Quit (Ping timeout: 480 seconds)
[20:35] * alram (~alram@ has joined #ceph
[20:35] * nwat (~nwat@eduroam-237-79.ucsc.edu) has joined #ceph
[20:37] * angdraug (~angdraug@c-98-248-39-148.hsd1.ca.comcast.net) Quit (Quit: Leaving)
[20:39] <swinchen> when adding a node you do not run "ceph-deploy new new-node-name" do you? you start by calling "ceph-deploy install ... "?
[20:41] * Tamil1 (~Adium@cpe-108-184-71-119.socal.res.rr.com) Quit (Quit: Leaving.)
[20:41] <cmdrk> hmm.. so i just made a very large ceph cluster with 58 OSDs, recreated the data, metadata, and rbd pools to have 2900 pgs each (50 * # OSDs) .. but all of my pgs are stuck unclean. did I miss something?
[20:42] * wschulze (~wschulze@cpe-72-229-37-201.nyc.res.rr.com) Quit (Quit: Leaving.)
[20:43] * Tamil1 (~Adium@cpe-108-184-71-119.socal.res.rr.com) has joined #ceph
[20:44] <ishkabob> cmdrk: the formula applies to ALL the pages in your cluster
[20:45] <ishkabob> so you have 2900 pages to distribute amongst all your pools
[20:45] <ishkabob> also, 2900 would only make sense if you are using a replication size of 2
[20:45] <ishkabob> the default is 3
[20:45] <swinchen> I am tryin to add a node and it gets stuck here: http://pastie.org/pastes/8339649/text No errors, no timeout.... nothing. 10+ minutes now.
[20:45] <ishkabob> (100*#OSDS)/ replicas
[20:46] <cmdrk> right, the plan was replication size of 2 -- either way, I'm not sure why the pgs are stuck unclean. i can recreate with replication size 3 etc.
[20:47] <ishkabob> cmdrk: your number of pgs is fine then, HOWEVER, that is the TOTAL number of pgs to distribute among your pools
[20:47] <ishkabob> so all the pages in your pools need to add up to 2900
[20:47] <cmdrk> right
[20:47] <ishkabob> you have too many pages in each pool, that being said, it should still recover (i think) eventually
[20:47] <ishkabob> you also need to match the pg_num to pgp_num
[20:48] <cmdrk> eh ill just recreate
[20:48] * markbby (~Adium@ has joined #ceph
[20:48] <ishkabob> swinchen: sounds like you're having auth problems, did you setup an ssh key for the other box?
[20:49] <infernix> librados: client.admin authentication error (1) Operation not permitted
[20:49] <infernix> what would cause this? I am an admin
[20:50] <swinchen> ishkabob: I added the key from admin node to ~ceph/.ssh/authorized_keys. Do I need to create keys on the new node and add them to the admin node?
[20:51] <ishkabob> infernix: client.admin is the name of the ceph admin user, it shoudl have an associated ceph keyring, are you using that keyring to auth?
[20:51] * markbby1 (~Adium@ Quit (Remote host closed the connection)
[20:51] <ishkabob> swinchen: on the "admin node" as whatever user you are running ceph-deploy with, run this "ssh ceph@ceph-test1"
[20:51] <ishkabob> does it prompt for a password?
[20:52] <swinchen> ishkabob: no, it logs me into the new node.
[20:52] * markbby1 (~Adium@ has joined #ceph
[20:52] <ishkabob> hmm, should be fine then, maybe not an auth problem
[20:52] <infernix> ishkabob: /etc/ceph/ceph.client.admin.keyring exists; should it not automatically use it?
[20:53] <ishkabob> infernix: it should, but its possible that the key wasn't imported into the ceph cluster when you created it, did you use ceph-deploy to create your cluster?
[20:54] <swinchen> hrmmm... not a problem with iptables.
[20:54] <ishkabob> swinchen: you could just install the ceph packages on your other machine manually
[20:54] <ishkabob> i actually haven't used ceph-deploy to install packages, i just use yum
[20:55] <cmdrk> ishkabob: so now after I recreate the pools (using ceph osd pool create [pool] pg_num pgp_num) I see all 100% of my placement groups in "active+degraded". there is no activity with ceph -w -- will these clear up on their own?
[20:55] <ishkabob> cmdrk: probably, did you delete the old pools?
[20:55] <cmdrk> i did
[20:55] <ishkabob> yeah, give it a few min
[20:56] <cmdrk> alright, thanks
[20:56] * markbby (~Adium@ Quit (Ping timeout: 480 seconds)
[20:56] <ishkabob> brb guys, lunch
[20:56] <infernix> ishkabob: yep. what would cause it not importing the key?
[20:57] * infernix runs a purge
[20:58] <swinchen> ishkabob: Interesting... I installed ceph-deploy on the new node, ran install and it worked fine. Go figure
[21:02] <swinchen> ugh, I am having bad luck today
[21:04] <infernix> admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
[21:04] <infernix> on the second mon in a mon create run
[21:04] <infernix> what the what
[21:07] <swinchen> so if I am trying to run "ceph-deploy mon create ceph-test1" and I get this output: http://pastie.org/pastes/8339699/text Where should I start looking? I ssh to ceph@ceph-test1 and sudo works fine...
[21:07] * swinchen is starting to dislike ceph-deploy
[21:10] * alfredo|noms is now known as alfredodeza
[21:12] <Gugge-47527> swinchen: are you running as root?
[21:13] <swinchen> Gugge-47527: yes... on the "admin" node.
[21:13] <Gugge-47527> it looks like the remote user does not have access to /etc/ceph ....
[21:14] <Gugge-47527> But if ceph-deploy ssh in as root, that is strange :)
[21:14] <alfredodeza> swinchen, Gugge-47527 all the ceph-deploy is doing really is just running commands that you would otherwise would have to do manually. There is little magic to it from that point of view
[21:14] <swinchen> Gugge-47527: the remote user is "ceph" I believe, but I setup passwordless sudo access (described in the quickstart guide)
[21:15] <alfredodeza> if something like "permission denied" comes up, you should really run that command that failed on the remote host and check what is happening
[21:15] <alfredodeza> swinchen: no, if you are executing as `root` you will log in as `root` on the remote host
[21:15] * gaveen (~gaveen@ Quit (Quit: Leaving)
[21:16] <alfredodeza> that is why that line in your output says that it will not use sudo (because it detected you logged in as root) [ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection without sudo
[21:16] <Gugge-47527> swinchen: how do you specify the remote user as ceph, when you run as root on the admin node?
[21:17] <swinchen> Gugge-47527: I add a "Host" entry to ~/.ssh/config
[21:17] <infernix> so weird
[21:17] <Gugge-47527> Dont do stuff like that :)
[21:17] <infernix> on my first mon in mon create A B C, things work fine
[21:17] <alfredodeza> aha
[21:17] <infernix> on my second
[21:17] <alfredodeza> so that is what is happening swinchen
[21:17] <infernix> 2013-09-19 15:17:03.756842 7f1e0ba7d780 -1 no public_addr or public_network specified, and mon.obj-bos-002 not present in monmap or ceph.conf
[21:17] <alfredodeza> you are specifying one user on your config but executing without sudo because you are root
[21:17] <alfredodeza> mistery solved
[21:17] <alfredodeza> don't do that :)
[21:17] <swinchen> Gugge-47527: Ok... that is what it says to do in the guide :/ hrmm... I think maybe I will start over and not use ceph-deploy
[21:18] <infernix> neither mon.A or mon.B is in ceph.conf, as ceph-deploy doesn't work that way
[21:18] <Gugge-47527> swinchen: what guide tells you to use different users on the admin node and the mon node?
[21:18] <Gugge-47527> That guide should be changed :)
[21:18] <alfredodeza> swinchen: does the guide mention you should alter your ssh config as the user you want to log in?
[21:18] <infernix> what happened to ceph-deploy >.< this used to work so well
[21:18] <alfredodeza> that would be a bug :(
[21:19] <swinchen> Gugge-47527: http://ceph.com/docs/master/start/quick-start-preflight/
[21:19] <alfredodeza> oh wow
[21:19] <alfredodeza> ok
[21:19] <alfredodeza> So I guess that is a bug because we should warn against executing as root
[21:20] * houkouonchi-datacenter (~linux@gw.sepia.ceph.com) has joined #ceph
[21:20] <Gugge-47527> swinchen: note that the guide does not say you should use root on the admin node :)
[21:20] <Gugge-47527> swinchen: but i agree, it should mention not to do that :)
[21:21] <swinchen> Gugge-47527: should I use the "ceph" user? or does it not matter as long as it isn't root?
[21:21] <Gugge-47527> it doesnt matter
[21:21] <Gugge-47527> but if you use the ceph user, you dont have to edit ssh config
[21:22] <swinchen> Ok, I am deleting the VMs and starting over.
[21:22] <infernix> well, something is hosed in ceph-deploy mon create
[21:23] <infernix> if i do mon create A B C, only A works, B and C don't
[21:23] * tziOm (~bjornar@ti0099a340-dhcp0395.bb.online.no) Quit (Remote host closed the connection)
[21:23] <infernix> if i do mon create A; mon create B; mon create C; all is well with $world
[21:23] <alfredodeza> created issue 6356
[21:23] <kraken> alfredodeza might be talking about: http://tracker.ceph.com/issues/6356 [docs should warn against executing ceph-deploy as root]
[21:23] <alfredodeza> swinchen: ^ ^
[21:23] <Gugge-47527> ceph-deploy is still new :)
[21:23] <infernix> argh
[21:24] <infernix> i purged, installed, did a new, created mon, did a gatherkeys, then admin my-admin-box
[21:24] <alfredodeza> infernix: do you have some output that we can at least look at?
[21:24] <infernix> and on admin box, ceph -w: librados: client.admin authentication error (1) Operation not permitted
[21:24] <swinchen> Gugge-47527: alfredodeza something I am unclear of from the guide... do I need to run "ceph-deploy new" on/for each node in the cluster? My guess is no... but I should check :)
[21:24] * twcai (~twcai@ Quit (Remote host closed the connection)
[21:24] <alfredodeza> infernix: we do test that scenario and it does work, I am curious to see your output and your ceph.conf after your failures
[21:25] <alfredodeza> infernix: you also need `sudo` for that `ceph -w` command
[21:25] <swinchen> haha! I think that is the 2nd bug posted today because of me :P
[21:25] <infernix> alfredodeza: i'm rootr
[21:25] <infernix> i will start over once more
[21:25] <alfredodeza> please do save the log output so I can take a look at them
[21:27] <Gugge-47527> swinchen: i dont remember what 'new' does :)
[21:27] <alfredodeza> Gugge-47527: that creates the ceph.conf file
[21:28] <infernix> alfredodeza: ohh. just noticed that 1 was running 0.67 and the 2 others 0.69
[21:28] <Gugge-47527> then it should only require the mons? :)
[21:28] <infernix> that's probably it
[21:28] * alfredodeza (~alfredode@c-24-131-46-23.hsd1.ga.comcast.net) has left #ceph
[21:31] * jskinner (~jskinner@ has joined #ceph
[21:34] * Tamil1 (~Adium@cpe-108-184-71-119.socal.res.rr.com) Quit (Quit: Leaving.)
[21:39] <infernix> that's better
[21:40] * angdraug (~angdraug@ has joined #ceph
[21:41] <infernix> note to self; do not mix dumpling with testing
[21:42] * Vjarjadian (~IceChat77@ has joined #ceph
[21:42] <infernix> Caution: invalid backup GPT header, but valid main header; regenerating backup header from main header.
[21:42] <infernix> Warning! Main and backup partition tables differ!
[21:43] <infernix> looks like zap-disk does not zap the backup of the GPT
[21:43] <infernix> iirc that's stored at the end of the disk?
[21:43] * scuttlemonkey (~scuttlemo@c-69-244-181-5.hsd1.mi.comcast.net) has joined #ceph
[21:43] * ChanServ sets mode +o scuttlemonkey
[21:45] * topro (~topro@host-62-245-142-50.customer.m-online.net) Quit (Remote host closed the connection)
[21:48] * TiCPU (~jeromepou@190-130.cgocable.ca) has joined #ceph
[21:48] * carif (~mcarifio@pool-96-233-32-122.bstnma.fios.verizon.net) has joined #ceph
[21:48] <TiCPU> I was wondering, is it possible to create multiple ceph filesystems for use with different data duplication levels? or should I use RBD in this case?
[21:50] <xarses> TiCPU, i think you have to create seperate ceph clusters to have different replication levels
[21:50] <xarses> but im not sure
[21:50] * Tamil1 (~Adium@cpe-108-184-71-119.socal.res.rr.com) has joined #ceph
[21:50] <TiCPU> nope, you can create multiple pools with different levels
[21:51] <TiCPU> but I don't know if multiple MDS can be specified different metadata/data pools
[21:53] <gregaf> sadly not
[21:54] * davidzlap (~Adium@cpe-75-84-249-188.socal.res.rr.com) Quit (Quit: Leaving.)
[21:54] <TiCPU> then I'll use RBD, thanks for the info, documentation wasn't explicit on that
[21:56] * nwat (~nwat@eduroam-237-79.ucsc.edu) Quit (Ping timeout: 480 seconds)
[21:57] * med is now known as pepegrigio
[21:58] <cmdrk> hrm. i did a "rados bench -p data 300 write" followed by a "rados bench -p data 300 rand" but i just get "Must write data before running a read benchmark!" :(
[21:58] * AaronSchulz_ (~chatzilla@ has joined #ceph
[21:59] <cmdrk> ah, maybe --no-cleanup
[22:00] <gregaf> and you'll need to do "seq" rather than rand; there isn't actually a rand benchmark implemented :D
[22:00] * nwat (~nwat@eduroam-237-79.ucsc.edu) has joined #ceph
[22:00] <cmdrk> :)
[22:00] <cmdrk> my pgs are still stuck in active+degraded :'(
[22:01] <cmdrk> created the pools about an hour ago now.
[22:02] * markbby1 (~Adium@ Quit (Quit: Leaving.)
[22:04] * AaronSchulz (~chatzilla@ Quit (Ping timeout: 480 seconds)
[22:04] * AaronSchulz_ is now known as AaronSchulz
[22:08] * Cube (~Cube@66-87-65-200.pools.spcsdns.net) has joined #ceph
[22:08] <TiCPU> cmdrk, active+degraded isn't going to get anywhere, it should be +recovering or +backfilling or such
[22:09] <swinchen> [remote] sudo: sorry, you must have a tty to run sudo <---- from ceph-deploy
[22:09] <TiCPU> swinchen, sudo is tying to ask for a password
[22:10] <TiCPU> /etc/sudoers should have a NOPASSWD: entry for ceph
[22:10] <swinchen> TiCPU: it does.
[22:10] * jeff-YF (~jeffyf@ Quit (Quit: jeff-YF)
[22:10] * jskinner (~jskinner@ Quit (Remote host closed the connection)
[22:10] <swinchen> ceph ALL = (root) NOPASSWD:ALL
[22:11] <swinchen> Defaults requiretty <---- ahhhh
[22:12] <TiCPU> didn't even know this option existed!
[22:12] <swinchen> That worked
[22:13] <swinchen> TiCPU: That was in the sudoers file
[22:13] <TiCPU> yes, I wonder how it is useful
[22:13] * BManojlovic (~steki@fo-d- has joined #ceph
[22:17] <cmdrk> TiCPU: any idea how to get them out of that state? these are freshly created pools.
[22:17] <cmdrk> i dont mind recreating the pools again, but the placement groups go directly into active+degraded after creation
[22:18] <swinchen> Hrmmm, are there any benefits to running multiple metadata nodes?
[22:19] <Gugge-47527> swinchen: yes, but its not as tested, so not as stable :)
[22:20] * ScOut3R (~scout3r@4E5C2305.dsl.pool.telekom.hu) Quit (Read error: Connection reset by peer)
[22:20] * ScOut3R (~scout3r@4E5C2305.dsl.pool.telekom.hu) has joined #ceph
[22:22] * tziOm (~bjornar@ti0099a340-dhcp0395.bb.online.no) has joined #ceph
[22:22] * ScOut3R (~scout3r@4E5C2305.dsl.pool.telekom.hu) Quit (Remote host closed the connection)
[22:25] <swinchen> What the heck!? http://pastie.org/pastes/8339880/text
[22:25] * claenjoy (~leggenda@ Quit (Quit: Leaving.)
[22:30] <swinchen> Well I think I am about done with this for now.
[22:37] * Cube (~Cube@66-87-65-200.pools.spcsdns.net) Quit (Quit: Leaving.)
[22:41] * carif (~mcarifio@pool-96-233-32-122.bstnma.fios.verizon.net) Quit (Quit: Ex-Chat)
[22:42] <Gugge-47527> swinchen: ceph -s looks for ceph.conf and key in /etc/ceph
[22:43] <Gugge-47527> if you only have it in current dir, you need to tell it what to use
[22:43] <Gugge-47527> or push the conf to the admin node, and use sudo ceph -s :)
[22:46] <swinchen> Gugge-47527: I tried this too: http://pastie.org/pastes/8339913/text
[22:47] <Gugge-47527> you need the admin key too :)
[22:48] <Gugge-47527> "missing keyring, cannot use cephx for authentication"
[22:50] <Gugge-47527> use the ceph param to specify keyfile, and point it to ceph.client.admin.keyring, or copy ceph.client.admin.keyringto /etc/ceph
[22:52] <swinchen> Gugge-47527: that files is already in /etc/ceph
[22:53] <Gugge-47527> and you just choose to not read what i write? :)
[22:53] <Gugge-47527> (sudo)
[22:54] <Gugge-47527> run "ls /etc/ceph && sudo ceph -s"
[22:54] <Gugge-47527> and paste the output
[22:54] <swinchen> DOH! I wasn't using sudo
[22:55] <swinchen> It has been a long day. I am sorry.
[22:55] <Gugge-47527> :)
[22:56] <swinchen> And thank you of course!
[22:58] <xarses> what / where would the centos/rhel package for libapache2-mod-fastcgi?, mod_fcgid appears to not support FastCGIExternalServer
[22:59] * sagelap (~sage@ Quit (Ping timeout: 480 seconds)
[23:03] * pepegrigio is now known as med
[23:03] * ishkabob (~c7a82cc0@webuser.thegrebs.com) Quit (Quit: TheGrebs.com CGI:IRC)
[23:03] <swinchen> Well that is interesting... I created 4 OSDs but ceph -s says I have none.
[23:04] <xarses> swinchen: ceph-deploy osd create?
[23:04] <swinchen> xarses: yep
[23:05] <xarses> change your command line from create to activate (ceph-deploy osd activate ...)
[23:05] <xarses> what distro?
[23:06] <xarses> swinchen: ^^
[23:07] <swinchen> xarses: fedora 19
[23:07] <xarses> ya, I think i need to log a bug on that
[23:08] <swinchen> xarses: interesting.. activate didn't seem to do anything either. prepare doesn't work because the disks are already mounted.
[23:09] <xarses> hmm
[23:09] <xarses> can you share the command line?
[23:10] <swinchen> ceph-deploy osd activate ceph0:vdc ceph0:vdd ceph1:vdc ceph1:vdd
[23:11] <swinchen> replace activate with prepare and you get this: http://pastie.org/pastes/8339973/text
[23:12] * dmick (~dmick@2607:f298:a:607:122:7863:5ab6:d460) has joined #ceph
[23:13] <xarses> the whole disks are supposed to be owned by ceph?
[23:14] <swinchen> xarses: yes
[23:15] * Guest7188 (~a@ Quit (Quit: This computer has gone to sleep)
[23:15] <xarses> ps axu | grep ceph and df?
[23:15] <swinchen> xarses: I already purged and am trying again with prepare/activate. Sorry... that was probably stupid of me
[23:15] <xarses> ok
[23:15] * athrift_ (~nz_monkey@ Quit (Read error: Connection reset by peer)
[23:16] <xarses> it sounds like ceph started the osd's fine
[23:16] * athrift (~nz_monkey@ has joined #ceph
[23:16] <xarses> but probably that the osd's couldn't reach the mon
[23:16] <xarses> so double check your iptables, the mon needs 6789 open, and the osd's need 6800-7100 open
[23:19] * SvenPHX (~scarter@wsip-174-79-34-244.ph.ph.cox.net) Quit (Remote host closed the connection)
[23:20] <swinchen> xarses: that might have been it. I set the iptables while it was doing it the last osd... now I have 1 OSD listed with ceph -s. No idea how to add the other ones :.
[23:21] <swinchen> gotta run. Thanks for the help
[23:21] <xarses> you can service ceph -a stop && service ceph -a start
[23:21] <xarses> yw
[23:21] <swinchen> ok let me try that real quick
[23:23] * a (~a@ has joined #ceph
[23:23] <swinchen> xarses: http://pastie.org/8340005
[23:23] <swinchen> Gotta jet! thanks!
[23:23] * a is now known as Guest7205
[23:24] * b1tbkt (~b1tbkt@24-217-192-155.dhcp.stls.mo.charter.com) Quit (Remote host closed the connection)
[23:30] * b1tbkt (~b1tbkt@24-217-192-155.dhcp.stls.mo.charter.com) has joined #ceph
[23:30] * ScOut3R (~scout3r@4E5C2305.dsl.pool.telekom.hu) has joined #ceph
[23:31] <xarses> swinchen: your fsid's look all screwed up on osd.0, check that they match /etc/ceph/ceph.conf and admin-node:~/ceph.conf
[23:31] <xarses> if they dont, you might want to start all the nodes over
[23:35] * bandrus (~Adium@cpe-76-95-217-129.socal.res.rr.com) Quit (Quit: Leaving.)
[23:39] * doxavore (~doug@99-7-52-88.lightspeed.rcsntx.sbcglobal.net) has joined #ceph
[23:40] * tziOm (~bjornar@ti0099a340-dhcp0395.bb.online.no) Quit (Remote host closed the connection)
[23:41] * SvenPHX (~scarter@wsip-174-79-34-244.ph.ph.cox.net) has joined #ceph
[23:45] * SvenPHX (~scarter@wsip-174-79-34-244.ph.ph.cox.net) Quit (Remote host closed the connection)
[23:45] * mattt (~mattt@lnx1.defunct.ca) has joined #ceph
[23:46] <mattt> hi all, got a mon that's binding to, any idea why it's doing that?
[23:47] * SvenPHX (~scarter@wsip-174-79-34-244.ph.ph.cox.net) has joined #ceph
[23:47] <gregaf> dmick: is that the issue you had that joao spent some time troubleshooting?
[23:48] <mattt> http://tracker.ceph.com/issues/5804 ? :)
[23:49] <dmick> sort of
[23:49] <dmick> mine was flipping between and its real address
[23:49] <dmick> and it was 6803
[23:49] <dmick> but either one is mysterious I think
[23:53] <gregaf> but apparently joao isn't around right now, bummer
[23:54] <mattt> no worries
[23:55] <mattt> i just deleted it and re-added
[23:55] <mattt> if i see it again i'll re-open/create a ticket
[23:55] <dmick> I'm adding a big comment to 5804
[23:55] <dmick> and may reopen
[23:55] <dmick> in fact I will reopen
[23:55] <dmick> it seems to me this must be resulting from someone calling Messenger::bind() with an ip but no port, and bind() starting at 6800 and working up
[23:56] <dmick> I don't know how the mon could do that exactly yet
[23:56] <mattt> yeah, i triple checked configurations/hostnames/etc. and everything is in order
[23:56] <mattt> tried changing it in the map and injecting again to no avail
[23:57] <dmick> I think you have to get a map, fix it, stop all the mons, inject it into all of them, and then restart
[23:57] <mattt> tried :(
[23:57] <dmick> and you need to use -f when doing that
[23:57] <dmick> so you can see the error messages
[23:57] <mattt> ah, -f

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.