#ceph IRC Log

Index

IRC Log for 2016-09-20

Timestamps are in GMT/BST.

[0:01] * newbie (~kvirc@host217-114-156-249.pppoe.mark-itt.net) Quit (Ping timeout: 480 seconds)
[0:06] * derjohn_mob (~aj@46.189.28.39) has joined #ceph
[0:10] * Pieman (~mps@26XAAB1UQ.tor-irc.dnsbl.oftc.net) Quit ()
[0:10] * Sketchfile (~galaxyAbs@185.3.135.178) has joined #ceph
[0:12] * ntpttr (~ntpttr@192.55.54.40) Quit (Remote host closed the connection)
[0:14] * ntpttr (~ntpttr@134.134.137.75) has joined #ceph
[0:15] * vasu (~vasu@c-73-231-60-138.hsd1.ca.comcast.net) Quit (Quit: Leaving)
[0:16] * vata (~vata@207.96.182.162) Quit (Quit: Leaving.)
[0:25] * doppelgrau (~doppelgra@dslb-088-072-094-200.088.072.pools.vodafone-ip.de) Quit (Quit: doppelgrau)
[0:31] * haplo37 (~haplo37@199.91.185.156) Quit (Remote host closed the connection)
[0:40] * Sketchfile (~galaxyAbs@185.3.135.178) Quit ()
[0:55] * bene2 (~bene@nat-pool-bos-t.redhat.com) Quit (Quit: Konversation terminated!)
[0:56] * sphinxx (~circuser-@154.120.74.134) Quit (Ping timeout: 480 seconds)
[0:59] * salwasser1 (~Adium@2601:197:101:5cc1:50fe:8001:efc9:f75a) Quit (Quit: Leaving.)
[1:07] * stiopa (~stiopa@cpc73832-dals21-2-0-cust453.20-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[1:07] * pico (~colde@tsn77-247-182-246.dyn.nltelcom.net) has joined #ceph
[1:17] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) has joined #ceph
[1:19] * wes_dillingham (~wes_dilli@209-6-222-74.c3-0.hdp-ubr1.sbo-hdp.ma.cable.rcn.com) has joined #ceph
[1:26] * [0x4A6F]_ (~ident@p4FC2691B.dip0.t-ipconnect.de) has joined #ceph
[1:29] * [0x4A6F] (~ident@0x4a6f.user.oftc.net) Quit (Ping timeout: 480 seconds)
[1:29] * [0x4A6F]_ is now known as [0x4A6F]
[1:34] * pdrakeweb (~pdrakeweb@pool-98-118-150-184.bflony.fios.verizon.net) Quit (Remote host closed the connection)
[1:37] * pico (~colde@5AEAABRPH.tor-irc.dnsbl.oftc.net) Quit ()
[1:42] * oms101 (~oms101@p20030057EA018300C6D987FFFE4339A1.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[1:51] * oms101 (~oms101@p20030057EA01CF00C6D987FFFE4339A1.dip0.t-ipconnect.de) has joined #ceph
[1:52] * ntpttr_ (~ntpttr@134.134.137.75) has joined #ceph
[1:52] * ntpttr (~ntpttr@134.134.137.75) Quit (Remote host closed the connection)
[1:56] * toastyde1th (~toast@pool-71-255-253-39.washdc.fios.verizon.net) has joined #ceph
[1:58] * ntpttr_ (~ntpttr@134.134.137.75) Quit (Remote host closed the connection)
[2:00] * borei (~dan@216.13.217.230) Quit (Ping timeout: 480 seconds)
[2:03] * toastydeath (~toast@pool-71-255-253-39.washdc.fios.verizon.net) Quit (Ping timeout: 480 seconds)
[2:09] * wushudoin (~wushudoin@2601:646:8200:c9f0:2ab2:bdff:fe0b:a6ee) Quit (Ping timeout: 480 seconds)
[2:20] * lixiaoy1 (~lixiaoy1@shzdmzpr01-ext.sh.intel.com) has joined #ceph
[2:31] * salwasser (~Adium@2601:197:101:5cc1:2d92:96da:4bb8:4c6c) has joined #ceph
[2:33] * ph470m (~ph470m@74.50.197.36) has joined #ceph
[2:35] * lri (kebac@limbo.freebsd.fi) has joined #ceph
[2:36] <ph470m> This is productive for 340 people...
[2:38] * saltsa_ (~joonas@dsl-hkibrasgw1-58c018-65.dhcp.inet.fi) has joined #ceph
[2:38] * saltsa (~joonas@dsl-hkibrasgw1-58c018-65.dhcp.inet.fi) Quit (Read error: Connection reset by peer)
[2:40] * lri_ (kebac@limbo.freebsd.fi) Quit (Ping timeout: 480 seconds)
[2:41] * ph470m (~ph470m@74.50.197.36) Quit (Quit: Quit)
[2:49] * wes_dillingham (~wes_dilli@209-6-222-74.c3-0.hdp-ubr1.sbo-hdp.ma.cable.rcn.com) Quit (Quit: wes_dillingham)
[2:53] * salwasser (~Adium@2601:197:101:5cc1:2d92:96da:4bb8:4c6c) Quit (Quit: Leaving.)
[3:07] * Kurt (~Adium@2001:628:1:5:e460:44e2:5a4f:a9db) Quit (Read error: Connection reset by peer)
[3:07] * Kurt (~Adium@2001:628:1:5:e460:44e2:5a4f:a9db) has joined #ceph
[3:10] * Kurt1 (~Adium@2001:628:1:5:e460:44e2:5a4f:a9db) has joined #ceph
[3:10] * Kurt (~Adium@2001:628:1:5:e460:44e2:5a4f:a9db) Quit (Read error: Connection reset by peer)
[3:10] * jfaj_ (~jan@p4FD273F7.dip0.t-ipconnect.de) has joined #ceph
[3:16] * Kurt (~Adium@2001:628:1:5:e460:44e2:5a4f:a9db) has joined #ceph
[3:17] * jfaj (~jan@p4FC258C9.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[3:21] * derjohn_mob (~aj@46.189.28.39) Quit (Ping timeout: 480 seconds)
[3:21] * Kurt1 (~Adium@2001:628:1:5:e460:44e2:5a4f:a9db) Quit (Ping timeout: 480 seconds)
[3:28] * ntpttr (~ntpttr@134.134.137.75) has joined #ceph
[3:28] * wes_dillingham (~wes_dilli@209-6-222-74.c3-0.hdp-ubr1.sbo-hdp.ma.cable.rcn.com) has joined #ceph
[3:29] * yanzheng (~zhyan@118.116.115.254) has joined #ceph
[3:30] * treenerd_ (~gsulzberg@cpe90-146-148-47.liwest.at) Quit (Quit: treenerd_)
[3:33] * EinstCrazy (~EinstCraz@222.69.240.111) has joined #ceph
[3:37] * EinstCrazy (~EinstCraz@222.69.240.111) Quit (Remote host closed the connection)
[3:41] * sebastian-w (~quassel@212.218.8.138) Quit (Remote host closed the connection)
[3:41] * sebastian-w (~quassel@212.218.8.138) has joined #ceph
[3:42] * ntpttr (~ntpttr@134.134.137.75) Quit (Ping timeout: 480 seconds)
[3:52] * mhack (~mhack@24-151-36-149.dhcp.nwtn.ct.charter.com) Quit (Remote host closed the connection)
[3:58] * northrup (~northrup@75-146-11-137-Nashville.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[4:00] * davidzlap (~Adium@2605:e000:1313:8003:1129:2f16:42d8:1acb) Quit (Quit: Leaving.)
[4:15] * Architect (~Teddybare@5.153.233.58) has joined #ceph
[4:20] * wjw-freebsd (~wjw@smtp.digiware.nl) Quit (Ping timeout: 480 seconds)
[4:23] * vicente (~~vicente@125-227-238-55.HINET-IP.hinet.net) has joined #ceph
[4:36] * kefu_ (~kefu@114.92.125.128) has joined #ceph
[4:37] * sphinxx (~sphinxx@154.120.74.134) has joined #ceph
[4:45] * Architect (~Teddybare@635AAAPIC.tor-irc.dnsbl.oftc.net) Quit ()
[5:03] * xarses (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) has joined #ceph
[5:11] * tdb (~tdb@myrtle.kent.ac.uk) Quit (Ping timeout: 480 seconds)
[5:13] * sphinxx (~sphinxx@154.120.74.134) Quit (Ping timeout: 480 seconds)
[5:17] * Vacuum_ (~Vacuum@88.130.219.117) has joined #ceph
[5:24] * Vacuum__ (~Vacuum@88.130.214.192) Quit (Ping timeout: 480 seconds)
[5:24] * tdb (~tdb@myrtle.kent.ac.uk) has joined #ceph
[5:30] * yanzheng (~zhyan@118.116.115.254) Quit (Quit: ??????)
[5:33] * puvo (~MKoR@tor2r.ins.tor.net.eu.org) has joined #ceph
[5:35] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) Quit (Quit: neurodrone_)
[6:03] * puvo (~MKoR@9J5AAAU9T.tor-irc.dnsbl.oftc.net) Quit ()
[6:07] * ivve (~zed@cust-gw-11.se.zetup.net) has joined #ceph
[6:11] * walcubi (~walcubi@p5795B501.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[6:11] * walcubi (~walcubi@p5797AF6C.dip0.t-ipconnect.de) has joined #ceph
[6:35] * johnavp1989 (~jpetrini@pool-100-14-10-2.phlapa.fios.verizon.net) Quit (Ping timeout: 480 seconds)
[6:43] * northrup (~northrup@173.14.101.193) has joined #ceph
[6:43] * colde1 (~Lattyware@108.61.122.8) has joined #ceph
[6:47] * johnavp1989 (~jpetrini@pool-100-14-10-2.phlapa.fios.verizon.net) has joined #ceph
[6:47] <- *johnavp1989* To prove that you are human, please enter the result of 8+3
[6:53] * rdas (~rdas@121.244.87.116) has joined #ceph
[6:57] * kefu_ is now known as kefu|afk
[6:59] * KindOne (kindone@0001a7db.user.oftc.net) Quit (Read error: Connection reset by peer)
[7:00] * johnavp1989 (~jpetrini@pool-100-14-10-2.phlapa.fios.verizon.net) Quit (Ping timeout: 480 seconds)
[7:01] * northrup (~northrup@173.14.101.193) Quit (Quit: My MacBook has gone to sleep. ZZZzzz???)
[7:03] * northrup (~northrup@173.14.101.193) has joined #ceph
[7:03] * kefu|afk is now known as kefu_
[7:10] * KindOne (kindone@h239.134.30.71.dynamic.ip.windstream.net) has joined #ceph
[7:13] * ade (~abradshaw@46.189.67.235) has joined #ceph
[7:13] * ade (~abradshaw@46.189.67.235) Quit (Remote host closed the connection)
[7:13] * colde1 (~Lattyware@9J5AAAVAQ.tor-irc.dnsbl.oftc.net) Quit ()
[7:15] * TomasCZ (~TomasCZ@yes.tenlab.net) Quit (Quit: Leaving)
[7:18] * stiopa (~stiopa@cpc73832-dals21-2-0-cust453.20-2.cable.virginm.net) has joined #ceph
[7:18] * morse (~morse@supercomputing.univpm.it) Quit (Ping timeout: 480 seconds)
[7:23] * morse (~morse@supercomputing.univpm.it) has joined #ceph
[7:29] * stiopa (~stiopa@cpc73832-dals21-2-0-cust453.20-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[7:34] * janos (~messy@static-71-176-211-4.rcmdva.fios.verizon.net) has joined #ceph
[7:40] * janos_ (~messy@static-71-176-211-4.rcmdva.fios.verizon.net) Quit (Ping timeout: 480 seconds)
[7:41] * rakeshgm (~rakesh@106.51.28.220) Quit (Remote host closed the connection)
[7:55] <Flynn> Hello all, I???m trying to install a new ceph cluster (jewel 10.2.2). I followed the quick install procedure and added two OSD???s (backed by a journal partition each on an SSD). That goes well. But, after this, the PG???s won???t get created. They get stuck:
[7:55] <Flynn> 64 pgs are stuck inactive for more than 300 seconds
[7:55] <Flynn> Does anybody know what is going wrong here?
[7:56] * northrup (~northrup@173.14.101.193) Quit (Quit: My MacBook has gone to sleep. ZZZzzz???)
[8:03] * wes_dillingham (~wes_dilli@209-6-222-74.c3-0.hdp-ubr1.sbo-hdp.ma.cable.rcn.com) Quit (Quit: wes_dillingham)
[8:04] * Be-El (~blinke@nat-router.computational.bio.uni-giessen.de) has joined #ceph
[8:09] * TMM (~hp@dhcp-077-248-009-229.chello.nl) Quit (Quit: Ex-Chat)
[8:14] * neurodrone (~neurodron@162.243.191.67) Quit (Ping timeout: 480 seconds)
[8:18] * EinstCrazy (~EinstCraz@58.247.119.250) has joined #ceph
[8:19] <ronrib> they might belong to the original default pool, try deleting that pool
[8:27] * jermudgeon (~jhaustin@199.200.6.173) has joined #ceph
[8:29] * sleinen (~Adium@2001:620:0:2d:a65e:60ff:fedb:f305) has joined #ceph
[8:31] * jermudgeon (~jhaustin@199.200.6.173) Quit ()
[8:32] * jermudgeon (~jhaustin@199.200.6.173) has joined #ceph
[8:51] * donatas (~donatas@88-119-196-104.static.zebra.lt) has joined #ceph
[8:51] <donatas> guys, I have 3 monitors, single node is down, but client side has all 3 nodes listed in /etc/ceph/ceph.conf?
[8:52] <donatas> does it have any performance penalty?
[8:52] <kefu_> donatas, the client will pick a monitor at random. if it fails to connect to it, it will try again.
[8:53] * m8x (~user@182.150.27.112) has joined #ceph
[8:53] <kefu_> but it's just an one shot thing, after it gets connected, it's will not try hunting for a monitor anymore.
[8:53] * m8x (~user@182.150.27.112) has left #ceph
[8:54] <kefu_> donatas unless it gets disconnected or the connected mon has a high latency.
[8:54] <donatas> hm
[8:54] <donatas> any command to see latencies?
[8:54] <donatas> because tracking with external tools like sysdig doesn't report any latencies
[8:54] <donatas> except futex() -> 5s.
[8:55] <donatas> and behavior is strange, writing big file to cephfs (ceph-fuse) takes 40Mb/s, while reading 400Mb/s
[9:01] <donatas> by the way, could it be such slow because of bad crush rules?
[9:02] * karnan (~karnan@125.16.34.66) has joined #ceph
[9:03] * T1w (~jens@node3.survey-it.dk) has joined #ceph
[9:05] * kefu_ (~kefu@114.92.125.128) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[9:06] * kefu_ (~kefu@114.92.125.128) has joined #ceph
[9:07] * doppelgrau (~doppelgra@dslb-088-072-094-200.088.072.pools.vodafone-ip.de) has joined #ceph
[9:10] <Be-El> donatas: do you use ssd based journals?
[9:10] * sleinen (~Adium@2001:620:0:2d:a65e:60ff:fedb:f305) has left #ceph
[9:10] <donatas> Be-El: yes, they reside on the same disk as data
[9:10] <Be-El> so you have a ssd-only setup?
[9:10] * kefu_ (~kefu@114.92.125.128) Quit (Max SendQ exceeded)
[9:11] <donatas> oh, sorry I have such setup: NVME SSD as cache & journal and data reside on SATA disks
[9:11] * kefu_ (~kefu@114.92.125.128) has joined #ceph
[9:12] * garphy`aw is now known as garphy
[9:12] <Be-El> ok, in that case 40 mb/s are probably too slow
[9:12] <donatas> yeah..
[9:12] <Be-El> did you run some osd level benchmarks, e.g. 'ceph tell osd.XYZ bench' ?
[9:12] <donatas> yes, I tried, they report 200-400Mb/s
[9:13] <Be-El> it should give you some approximation about what write speed the osd is capable of
[9:13] <IcePic> sometimes, a single bad/slow disk can make large operations crawl
[9:13] <donatas> let me try to do this test again
[9:13] <Be-El> 400 mb/s is rather slow for nvme based journals
[9:14] <donatas> average is 200mb/s for all.. too much slow
[9:14] <Be-El> even if murphy strikes and you just hit the filestore flush window
[9:14] <Be-El> which kind of nvme ssd do you use?
[9:15] <donatas> https://gist.github.com/ton31337/efc553dfececc24b354c572c1c6fd2b9
[9:15] <donatas> I think too slow here..
[9:15] <T1w> I remember someone who had a ssd only setup - he got 2GB/s (yes gigabytes!) writes
[9:15] <Be-El> "bytes_per_sec": 861974667
[9:15] <donatas> it's your?
[9:15] <Be-El> that's a intel p3700 as journal
[9:16] <Be-El> and it's a small one (400gb); the large ones have more memory lanes and thus are faster
[9:17] * fsimonce (~simon@host98-71-dynamic.1-87-r.retail.telecomitalia.it) has joined #ceph
[9:18] <Be-El> donatas: did you ran the usual ssd benchmark on your journals (http://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/)
[9:18] <donatas> will try these
[9:20] * Hemanth (~hkumar_@125.16.34.66) has joined #ceph
[9:20] * nass5 (~fred@l-p-dn-in-12a.lionnois.site.univ-lorraine.fr) has joined #ceph
[9:21] * nass5 (~fred@l-p-dn-in-12a.lionnois.site.univ-lorraine.fr) Quit (Remote host closed the connection)
[9:22] * nass5 (~fred@l-p-dn-in-12a.lionnois.site.univ-lorraine.fr) has joined #ceph
[9:22] <donatas> Be-El: Intel Corporation DC P3700 SSD this is
[9:22] <donatas> we use
[9:23] <Be-El> donatas: ok, it's the same nvme device we use
[9:23] <donatas> Be-El: what configuration do you use? do you use it for journal+cache?
[9:23] <Be-El> donatas: your cluster should definitely be able to provide more throughput
[9:23] <Be-El> the ssds are currently only used for journals and the cephfs metadata pool
[9:24] <Be-El> cache tier is on the todo list
[9:25] <Be-El> you should probably focus on the individual osd write speed, since these ssds should give you much more throughput
[9:25] <Be-El> how many hosts do you have?
[9:25] * wjw-freebsd (~wjw@smtp.digiware.nl) has joined #ceph
[9:25] <donatas> I have 2 OSDs and 3 monitors
[9:26] * Flynn (~stefan@ip-185-87-117-140.fiber.nl) Quit (Quit: Flynn)
[9:26] <Be-El> how many hdds per host?
[9:26] <donatas> 12 disks
[9:27] <donatas> https://gist.github.com/ton31337/b938186b2d14f6ebe0d7a6e48685081f
[9:27] <donatas> ~200mb/s average on both OSDs nodes
[9:27] * northrup (~northrup@173.14.101.193) has joined #ceph
[9:28] <Be-El> i'm not sure whether the osd benchmark also involves the data replication
[9:29] <donatas> hm
[9:29] <Be-El> donatas: http://paste.debian.net/830302/
[9:30] <Be-El> donatas: that's our setup for osds, taken from various tuning guides and put together without much testing (and knowing what it actually does ;-) )
[9:30] <donatas> let me try this ;-)
[9:30] <Be-El> donatas: especially the disabling of the wbthrottle might not be the best idea
[9:30] <Be-El> try this for _one_ osd by adding to to ceph.conf, restart the osd, and run osd bench for it
[9:31] * kefu_ (~kefu@114.92.125.128) Quit (Max SendQ exceeded)
[9:31] <donatas> Be-El: great, let's do it
[9:31] <Be-El> and it's for the jewel release, if you a running on hammer you might need to skip some of the settings, e.g. osd_op_queue
[9:31] * kefu_ (~kefu@114.92.125.128) has joined #ceph
[9:32] <donatas> i'm running jewel
[9:35] * vicente_ (~~vicente@125-227-238-55.HINET-IP.hinet.net) has joined #ceph
[9:35] * vicente (~~vicente@125-227-238-55.HINET-IP.hinet.net) Quit (Read error: Connection reset by peer)
[9:38] <donatas> Be-El: just tried with your setup, nothing changed in numbers (ceph tell osd..)
[9:39] <donatas> could it be the problem, that our metadata sits on system disks, not in nvme?
[9:39] <Be-El> metadata as the metadata pool?
[9:39] <Be-El> or metadata as the mon/osd metadata?
[9:40] <donatas> metadata pool
[9:41] <Be-El> that shouldn't matter, since 'ceph tell ...' works on osds itself, not on pool level
[9:42] <Be-El> how much ram and cpus do your hosts have?
[9:44] <donatas> Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
[9:44] <donatas> 64GB memory
[9:44] <donatas> OSDs
[9:47] <Be-El> ram should be ok (although more ram is always a good idea), but i'm not sure about the cpu
[9:47] * branto (~branto@178.253.167.12) has joined #ceph
[9:47] <Be-El> i would propose to run system monitoring tools like atop or nmon while running the benchmarks. they might give you hint on potential bottlenecks
[9:48] * vicente_ (~~vicente@125-227-238-55.HINET-IP.hinet.net) Quit (Read error: Connection reset by peer)
[9:48] * vicente (~~vicente@125-227-238-55.HINET-IP.hinet.net) has joined #ceph
[9:49] * derjohn_mob (~aj@p578b6aa1.dip0.t-ipconnect.de) has joined #ceph
[9:49] * northrup (~northrup@173.14.101.193) Quit (Quit: Textual IRC Client: www.textualapp.com)
[9:53] * EinstCrazy (~EinstCraz@58.247.119.250) Quit (Remote host closed the connection)
[9:53] <donatas> Be-El: OSDs and MON hosts are almost idle :/
[9:54] * dgurtner (~dgurtner@84-73-130-19.dclient.hispeed.ch) has joined #ceph
[9:55] <donatas> Be-El: could you please try to extract linux kernel source under cephfs mount?
[9:55] <donatas> Be-El: by the way do you use ceph-fuse or kernel mount?
[9:55] <Be-El> kernel based mounts
[9:57] * nils_ (~nils_@doomstreet.collins.kg) has joined #ceph
[9:57] <donatas> real 7m4.860s
[9:57] * yanzheng (~zhyan@118.116.115.254) has joined #ceph
[9:57] <donatas> Be-El: which distro/kernel do you use?
[9:58] * Lokta (~Lokta@carbon.coe.int) has joined #ceph
[9:58] <Be-El> donatas: centos 7.2 with standard kernel on the ceph hosts, ubuntu with kernel 4.6.6 on clients
[9:59] * DanFoster (~Daniel@office.34sp.com) has joined #ceph
[10:02] <donatas> hm, that's why I can't use kernel mount with centos 7.2 on clients
[10:02] <donatas> or is it possible?
[10:02] <donatas> at the moment i'm using ceph-fuse
[10:02] <Be-El> the standard kernel on centos is probably too old
[10:02] <donatas> yeah
[10:02] <Be-El> you can try a more recent one from the elrepo repository
[10:03] <Be-El> but your problem is not ceph-fuse; the osds are not performing well
[10:03] * koma (~koma@219.95.168.109.host.static.ip.kpnqwest.it) has joined #ceph
[10:03] <donatas> Be-El: but OSD drives itself have maximum as 200MB/s
[10:03] <donatas> physically
[10:03] <donatas> or I don't understand this test
[10:04] <Be-El> afaik the test uses the current osd setup to write a bunch of data. it should include the journals etc.
[10:04] * ledgr (~ledgr@88-119-196-104.static.zebra.lt) has joined #ceph
[10:04] <donatas> Be-El: what's your rsults from osd tell?
[10:05] <Be-El> "bytes_per_sec": 961368698 (hdd osd)
[10:05] <Be-El> "bytes_per_sec": 1029714440 (ssd osd)
[10:05] <Be-El> both are using journals on the nvme drives
[10:05] <donatas> what kind of hdd drives?
[10:06] <Be-El> nl-sas drives rebranded by HP
[10:06] <Be-El> 7.2rpm, 6/4 tb
[10:07] <Be-El> so the bench command does not give you an estimation of the drive speed, but the journal speed
[10:07] <donatas> got it
[10:08] <Be-El> you have cache pools; what's the throughput of them if you test one of them with ceph tell osd.X bench?
[10:08] <ledgr> We have 12xHDD that are using journals on 1xNVME (splitted to 65GiB partitions)
[10:08] <donatas> Be-El: "bytes_per_sec": 1058693348
[10:09] <donatas> too little I think
[10:09] <Be-El> that's 1gb/s. that's as fast as possible
[10:10] <donatas> not 2GB/s?
[10:10] <donatas> it's locally connected
[10:10] <ledgr> Based on our 'bench' results I have suspicion that our journals are not working?
[10:10] <ledgr> How do we check?
[10:10] * binary7 (~antanas@88-119-196-104.static.zebra.lt) has joined #ceph
[10:11] <Be-El> journals writes are synchronous writes; the benchmark from sebastien han's blog give me about 1.2 gb/s for our p3700. so you are more or less maxing out the device
[10:11] <ledgr> Be-El: I mean you have 900MB/s for HDD with NVME journal and we have only 200MB/s with NVME journal
[10:12] <Be-El> ledgr: check that the journal symlink in /var/lib/ceph/osd/ceph-XYZ points to the correct partition
[10:13] <ledgr> There is: lrwxrwxrwx 1 ceph ceph 14 Sep 11 22:29 journal -> /dev/nvme0n1p1
[10:14] <ledgr> blkid /dev/nvme0n1p1
[10:14] <ledgr> /dev/nvme0n1p1: PARTLABEL="Linux filesystem" PARTUUID="3d849b65-364f-4039-a533-95c830782d55"
[10:15] <ledgr> Does it matter if journal partition is marked as 'Linux filesystem' and not 'ceph journal'?
[10:15] <Be-El> that's just a label. as long as the partition type uuid is correct
[10:15] <donatas> [root@us-imm-cephmon1 ~]# blkid /dev/nvme0n1p2
[10:15] <donatas> /dev/nvme0n1p2: PARTLABEL="ceph journal" PARTUUID="7792dd08-6654-445f-96b9-eb865b835b66"
[10:15] * TMM (~hp@ip-80-113-15-100.ip.prioritytelecom.net) has joined #ceph
[10:15] <donatas> it's p2, not p1
[10:15] <donatas> :)
[10:18] * derjohn_mob (~aj@p578b6aa1.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[10:19] <ledgr> So we have a situation here:
[10:19] <ledgr> journal -> /dev/nvme0n1p1
[10:19] <donatas> ledgr: /dev/nvme0n1p2 !
[10:19] <ledgr> This is for osd.13 :)
[10:20] <ledgr> lrwxrwxrwx 1 root root 15 Sep 20 08:14 /dev/disk/by-partuuid/3d849b65-364f-4039-a533-95c830782d55 -> ../../nvme0n1p1
[10:20] <ledgr> cat journal_uuid
[10:20] <ledgr> 6d9add0a-7845-466c-ae15-9d5830a0e285
[10:20] <donatas> ok
[10:20] <ledgr> journal_uuid content is different from partition ID to which journal is symlinked to
[10:21] <ledgr> Everything was set up using ceph-deploy
[10:21] <Be-El> the journal setup seems to be ok
[10:22] <Be-El> you can also check the osd logs in /var/log/ceph. it should indicate which journal is used (size, journaling mode etc)
[10:22] <donatas> 2016-09-12 08:41:23.888702 7f4f00d7e800 1 journal _open /dev/nvme0n1p2 fd 4: 21474836480 bytes, block size 4096 bytes, directio = 0, aio = 0
[10:22] <donatas> looks ok too ;-)
[10:23] <ledgr> Be-El: journal_uuid contains partition UUID that does not even exist in that host
[10:25] <Be-El> ledgr: if you refer to /dev/disk/by-uuid, the udev rule for journals partitions might prevent the symlink on that directory from being created
[10:27] * Flynn (~stefan@84.241.194.212) has joined #ceph
[10:28] * malevolent (~quassel@192.146.172.118) has joined #ceph
[10:31] * ade (~abradshaw@212095007207.public.telering.at) has joined #ceph
[10:36] * malevolent (~quassel@192.146.172.118) Quit (Remote host closed the connection)
[10:37] * fdmanana (~fdmanana@2001:8a0:6e0c:6601:35fa:21a0:3929:f94d) has joined #ceph
[10:38] * hyperbaba (~hyperbaba@private.neobee.net) has joined #ceph
[10:38] * malevolent (~quassel@192.146.172.118) has joined #ceph
[10:38] <hyperbaba> Hi there, can somebody help me with the problem of incomplete pgs?
[10:38] * sphinxx (~sphinxx@154.120.74.134) has joined #ceph
[10:40] * Dw_Sn (~Dw_Sn@00020a72.user.oftc.net) has joined #ceph
[10:40] * binary7 (~antanas@88-119-196-104.static.zebra.lt) Quit (Quit: leaving)
[10:43] * sudocat (~dibarra@cpe-76-173-12-203.hawaii.res.rr.com) has joined #ceph
[10:46] * rakeshgm (~rakesh@125.16.34.66) has joined #ceph
[10:51] * sudocat (~dibarra@cpe-76-173-12-203.hawaii.res.rr.com) Quit (Ping timeout: 480 seconds)
[10:54] <hyperbaba> i had some wierd crash and out of 3 osd disks. The 6 placement groups are in incomplete state on the cluster. I've checked the dirs for that pgs and they where empty on active osd-s and started to fill up with the new data... how can i import data from old disks ( i did pgexport on them and i have export file) The problem is that the recepie said that new pg dirs needs to be removed from the acting osd, so i will endup in incomplete state again if i
[10:54] <hyperbaba> fill it with the export data.
[10:57] * nils_ (~nils_@doomstreet.collins.kg) Quit (Quit: This computer has gone to sleep)
[10:58] * derjohn_mob (~aj@46.189.28.39) has joined #ceph
[11:01] * lixiaoy1 (~lixiaoy1@shzdmzpr01-ext.sh.intel.com) Quit (Ping timeout: 480 seconds)
[11:03] <donatas> Be-El: thank you, problem solved.
[11:03] <donatas> there were bad symlinks to journal partitions
[11:06] <peetaur2> hyperbaba: why don't you just add the old disks back as they were, and then set them out one at a time, replace, etc.?
[11:06] * Dw_Sn (~Dw_Sn@00020a72.user.oftc.net) Quit (Ping timeout: 480 seconds)
[11:12] * TMM (~hp@ip-80-113-15-100.ip.prioritytelecom.net) Quit (Quit: Ex-Chat)
[11:13] * bara (~bara@nat-pool-brq-t.redhat.com) has joined #ceph
[11:16] * arthur (~arthur@89.200.203.7) has joined #ceph
[11:17] * Flynn (~stefan@84.241.194.212) Quit (Quit: Flynn)
[11:17] * doppelgrau (~doppelgra@dslb-088-072-094-200.088.072.pools.vodafone-ip.de) Quit (Quit: doppelgrau)
[11:23] * bauruine (~bauruine@mail.tuxli.ch) Quit (Quit: ZNC - http://znc.in)
[11:23] * bauruine (~bauruine@mail.tuxli.ch) has joined #ceph
[11:34] * fdmanana (~fdmanana@2001:8a0:6e0c:6601:35fa:21a0:3929:f94d) Quit (Ping timeout: 480 seconds)
[11:35] * ircuser-1 (~Johnny@158.183-62-69.ftth.swbr.surewest.net) Quit (Ping timeout: 480 seconds)
[11:36] <hyperbaba> peetaur: the problem is that they where set down and out
[11:36] <hyperbaba> peetaur2: the problem is that they where set down and out
[11:36] <hyperbaba> how can i add them back in if they are down and out
[11:37] * fdmanana (~fdmanana@2001:8a0:6e0c:6601:35fa:21a0:3929:f94d) has joined #ceph
[11:38] <peetaur2> start them and they're not down, then set them in and they're not out
[11:39] <hyperbaba> i can't start the osd process on them it fails right after start....
[11:39] <hyperbaba> how to "unset" out?
[11:40] <peetaur2> ceph osd in <ids> [<ids>...]
[11:43] <peetaur2> and to make them start, let's see the log or output when running with -f
[11:44] * TMM (~hp@ip-80-113-15-100.ip.prioritytelecom.net) has joined #ceph
[11:45] <hyperbaba> the problem is that coleque of mine removed auth too for those osd-s
[11:45] <hyperbaba> and they are showing in the crushmap as devicex not osd
[11:46] * arthur (~arthur@89.200.203.7) has left #ceph
[11:46] <hyperbaba> how can i restore auth
[11:46] <hyperbaba> for those osd?
[11:49] * ivve (~zed@cust-gw-11.se.zetup.net) Quit (Ping timeout: 480 seconds)
[11:52] * rakeshgm (~rakesh@125.16.34.66) Quit (Quit: Peace :))
[11:56] <hyperbaba> he removed all those osd-s from the system
[11:56] <hyperbaba> damn
[11:56] <hyperbaba> how can i recover data
[11:56] <hyperbaba> i am currently making exports of those affected osds using tools from ceph-test....can i "merge" that data with existing pg somehow?
[11:58] * doppelgrau (~doppelgra@132.252.235.172) has joined #ceph
[12:03] <peetaur2> ok restoring auth is easy I think
[12:03] <peetaur2> first check the situation: ceph auth list (this will display all keys on screen...so close your windows and check behind you for spies)
[12:04] <peetaur2> then add them back like this: cat /var/lib/ceph/osd/{cluster}-{osdnum}/keyring to see it, then ceph auth add osd.{osdnum} -i /path/to/keyring
[12:04] <peetaur2> be very very very careful not to change old users, esp. mons and admin client..you can accidently lock out admin and all daemons from everything
[12:06] * derjohn_mob (~aj@46.189.28.39) Quit (Ping timeout: 480 seconds)
[12:06] <peetaur2> and then ceph auth caps osd.{osdnum} mon "...." osd "..." replace the ... with what you saw for other osds in the ceph auth list. for me this is ceph auth caps osd.0 mon "allow profile osd" osd "allow *"
[12:07] <peetaur2> "caps" is probably more dangerous than "add"... I think add won't break an old user.
[12:07] <peetaur2> oh and you can combine caps into the add command
[12:07] <frickler> I'm seeing this sporadic error from python-rbd, can anyone help? https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1625489
[12:08] <hyperbaba> but he also removed osd from the crush map
[12:08] <hyperbaba> is there a help for that?
[12:08] <peetaur2> hyperbaba: yes you can add it back to the crush map
[12:09] <peetaur2> all I'm unsure of is whether the files on disk are reused simply, or if there is more work
[12:09] <peetaur2> I have not tried damaging an osd and re-adding it
[12:09] <peetaur2> btw, for best results, reproduce this problem on a test cluster and do the same steps to test them...
[12:10] <peetaur2> oh and maybe you need the old osd fsid ...no idea how to get that if you lost it
[12:10] <peetaur2> do you have to add an osd, or just add it to the crush map?
[12:11] <peetaur2> so I'm not yet an expert, so I'll just say again, make sure to test on a test cluster to know if this is something that will further damage the data
[12:11] <hyperbaba> i really don't know what do i need. i have 6 pg in incomplete state and data fro those pgs is on osd-s which where outed and removed from the cluster (auth and crushmap) . I just wan't to merge this data to the cluster
[12:12] * karnan (~karnan@125.16.34.66) Quit (Ping timeout: 480 seconds)
[12:15] <peetaur2> I thought you just simply pulled disks or killed the daemons and so it would be easy to add again
[12:15] * derjohn_mob (~aj@46.189.28.53) has joined #ceph
[12:15] <peetaur2> but if you have deleted the crush map, etc. then I am not really sure if simply adding them again works
[12:15] <hyperbaba> that's true
[12:16] * TMM (~hp@ip-80-113-15-100.ip.prioritytelecom.net) Quit (Quit: Ex-Chat)
[12:17] <peetaur2> but I'm sure adding auth again is fine...just not sure about the other steps like:
[12:17] <peetaur2> ceph osd create <osd uuid>
[12:17] <peetaur2> ceph osd crush add-bucket ...
[12:17] <peetaur2> ceph osd crush move ...
[12:17] <peetaur2> ceph osd crush add osd.{osd number} ...
[12:18] * adamcrume_ (~quassel@2601:647:cb01:f890:a288:69ff:fe70:6caa) Quit (Ping timeout: 480 seconds)
[12:19] <peetaur2> for the details see http://docs.ceph.com/docs/master/install/manual-deployment/#long-form
[12:19] * ade (~abradshaw@212095007207.public.telering.at) Quit (Remote host closed the connection)
[12:24] * malevolent (~quassel@192.146.172.118) Quit (Ping timeout: 480 seconds)
[12:24] * karnan (~karnan@125.16.34.66) has joined #ceph
[12:25] * sphinxx (~sphinxx@154.120.74.134) Quit (Ping timeout: 480 seconds)
[12:25] * malevolent (~quassel@192.146.172.118) has joined #ceph
[12:30] * vicente (~~vicente@125-227-238-55.HINET-IP.hinet.net) Quit (Quit: Leaving)
[12:31] * TMM (~hp@185.5.121.201) has joined #ceph
[12:40] * karnan (~karnan@125.16.34.66) Quit (Ping timeout: 480 seconds)
[12:43] * jcsp (~jspray@82-71-16-249.dsl.in-addr.zen.co.uk) has joined #ceph
[12:44] * ade (~abradshaw@212095007207.public.telering.at) has joined #ceph
[12:46] * newbie (~kvirc@host217-114-156-249.pppoe.mark-itt.net) has joined #ceph
[12:46] * schegi (~schegi@81.169.147.212) Quit (Quit: leaving)
[12:47] * schegi (~schegi@81.169.147.212) has joined #ceph
[12:50] * kefu_ (~kefu@114.92.125.128) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[12:50] * schegi (~schegi@81.169.147.212) Quit ()
[12:50] * schegi (~schegi@81.169.147.212) has joined #ceph
[12:52] * kefu_ (~kefu@114.92.125.128) has joined #ceph
[12:52] * donatas (~donatas@88-119-196-104.static.zebra.lt) Quit (Quit: leaving)
[12:52] * karnan (~karnan@125.16.34.66) has joined #ceph
[12:53] * garphy is now known as garphy`aw
[12:56] * garphy`aw is now known as garphy
[12:58] * ivve (~zed@cust-gw-11.se.zetup.net) has joined #ceph
[13:03] * bniver (~bniver@71-9-144-29.static.oxfr.ma.charter.com) has joined #ceph
[13:06] * wjw-freebsd (~wjw@smtp.digiware.nl) Quit (Ping timeout: 480 seconds)
[13:09] * EinstCrazy (~EinstCraz@183.195.124.35) has joined #ceph
[13:09] * rdias (~rdias@2001:8a0:749a:d01:f539:9e93:3054:d071) Quit (Ping timeout: 480 seconds)
[13:10] * derjohn_mob (~aj@46.189.28.53) Quit (Ping timeout: 480 seconds)
[13:13] * derjohn_mob (~aj@46.189.28.53) has joined #ceph
[13:14] * topro (~prousa@p578af414.dip0.t-ipconnect.de) Quit (Remote host closed the connection)
[13:15] * branto (~branto@178.253.167.12) Quit (Ping timeout: 480 seconds)
[13:15] * ade (~abradshaw@212095007207.public.telering.at) Quit (Ping timeout: 480 seconds)
[13:17] * bene2 (~bene@nat-pool-bos-t.redhat.com) has joined #ceph
[13:17] * karnan (~karnan@125.16.34.66) Quit (Ping timeout: 480 seconds)
[13:20] * binary7 (~antanas@88-119-196-104.static.zebra.lt) has joined #ceph
[13:20] * Flynn (~stefan@84.241.194.212) has joined #ceph
[13:21] * topro (~prousa@p578af414.dip0.t-ipconnect.de) has joined #ceph
[13:23] * Flynn (~stefan@84.241.194.212) has left #ceph
[13:25] * neurodrone (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) has joined #ceph
[13:26] * karnan (~karnan@125.16.34.66) has joined #ceph
[13:30] * branto (~branto@178.253.167.12) has joined #ceph
[13:30] * branto (~branto@178.253.167.12) Quit ()
[13:31] * branto (~branto@178.253.167.12) has joined #ceph
[13:39] * neurodrone (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) Quit (Quit: neurodrone)
[13:40] * salwasser (~Adium@2601:197:101:5cc1:cae0:ebff:fe18:8237) has joined #ceph
[13:45] * salwasser (~Adium@2601:197:101:5cc1:cae0:ebff:fe18:8237) Quit (Quit: Leaving.)
[13:45] * jermudgeon (~jhaustin@199.200.6.173) Quit (Quit: jermudgeon)
[13:45] * ade (~abradshaw@we-marry-in.berlin) has joined #ceph
[13:45] * ade (~abradshaw@we-marry-in.berlin) Quit (Remote host closed the connection)
[13:46] * ade_b (~abradshaw@we-marry-in.berlin) has joined #ceph
[13:46] * ade_b (~abradshaw@we-marry-in.berlin) Quit ()
[13:49] * rdias (~rdias@bl7-92-98.dsl.telepac.pt) has joined #ceph
[13:51] * kefu_ (~kefu@114.92.125.128) Quit (Max SendQ exceeded)
[13:52] * kefu (~kefu@114.92.125.128) has joined #ceph
[13:52] * karnan (~karnan@125.16.34.66) Quit (Ping timeout: 480 seconds)
[13:55] * valeech (~valeech@166.170.31.125) has joined #ceph
[13:55] * karnan (~karnan@125.16.34.66) has joined #ceph
[13:56] * binary7 (~antanas@88-119-196-104.static.zebra.lt) Quit (Quit: leaving)
[14:01] * EinstCrazy (~EinstCraz@183.195.124.35) Quit (Ping timeout: 480 seconds)
[14:01] * kefu (~kefu@114.92.125.128) Quit (Max SendQ exceeded)
[14:02] * kefu (~kefu@45.76.99.99) has joined #ceph
[14:03] * pdrakeweb (~pdrakeweb@pool-98-118-150-184.bflony.fios.verizon.net) has joined #ceph
[14:06] * EinstCrazy (~EinstCraz@183.195.124.35) has joined #ceph
[14:12] * karnan (~karnan@125.16.34.66) Quit (Ping timeout: 480 seconds)
[14:16] * dneary (~dneary@nat-pool-bos-u.redhat.com) has joined #ceph
[14:18] * neurodrone (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) has joined #ceph
[14:21] * arbrandes (~arbrandes@ec2-54-172-54-135.compute-1.amazonaws.com) has joined #ceph
[14:24] * mattbenjamin (~mbenjamin@76-206-42-50.lightspeed.livnmi.sbcglobal.net) has joined #ceph
[14:24] * karnan (~karnan@125.16.34.66) has joined #ceph
[14:26] * rdas (~rdas@121.244.87.116) Quit (Quit: Leaving)
[14:28] * arbrandes1 (~arbrandes@ec2-54-172-54-135.compute-1.amazonaws.com) Quit (Ping timeout: 480 seconds)
[14:32] * kefu_ (~kefu@114.92.125.128) has joined #ceph
[14:33] * Nicho1as (~nicho1as@00022427.user.oftc.net) has joined #ceph
[14:33] * rraja (~rraja@125.16.34.66) has joined #ceph
[14:34] <mistur> hello
[14:35] <mistur> in jewel how I can change the default data pool of cephfs ?
[14:35] <mistur> I create a new pool, add to the fs
[14:35] <mistur> I want to delete the old one
[14:36] * kefu_ is now known as kefu|afk
[14:36] * kefu|afk is now known as kefu_
[14:36] <mistur> but I got Error EINVAL: cannot remove default data pool
[14:38] * kefu (~kefu@45.76.99.99) Quit (Ping timeout: 480 seconds)
[14:42] * Racpatel (~Racpatel@2601:87:3:31e3::77ec) has joined #ceph
[14:42] * kefu_ is now known as kefu|afk
[14:43] * valeech (~valeech@166.170.31.125) Quit (Quit: valeech)
[14:46] * newbie (~kvirc@host217-114-156-249.pppoe.mark-itt.net) Quit (Read error: No route to host)
[14:47] * newbie (~kvirc@host217-114-156-249.pppoe.mark-itt.net) has joined #ceph
[14:48] * rakeshgm (~rakesh@125.16.34.66) has joined #ceph
[14:51] * donatas (~donatas@88-119-196-104.static.zebra.lt) has joined #ceph
[14:51] * johnavp1989 (~jpetrini@8.39.115.8) has joined #ceph
[14:51] <- *johnavp1989* To prove that you are human, please enter the result of 8+3
[14:53] * mhack (~mhack@nat-pool-bos-u.redhat.com) has joined #ceph
[14:54] * ivve (~zed@cust-gw-11.se.zetup.net) Quit (Ping timeout: 480 seconds)
[14:55] * ivve (~zed@cust-gw-11.se.zetup.net) has joined #ceph
[14:59] * bara (~bara@nat-pool-brq-t.redhat.com) Quit (Ping timeout: 480 seconds)
[15:01] <Be-El> mistur: you cannot remove the first data pool since it contains some of the metadata
[15:07] * kefu|afk (~kefu@114.92.125.128) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[15:08] * dugravot61 (~dugravot6@nat-persul-plg.wifi.univ-lorraine.fr) has joined #ceph
[15:09] * neurodrone (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) Quit (Quit: neurodrone)
[15:10] * dugravot6 (~dugravot6@l-p-dn-in-4a.lionnois.site.univ-lorraine.fr) Quit (Ping timeout: 480 seconds)
[15:10] * bara (~bara@213.175.37.12) has joined #ceph
[15:12] * rakeshgm (~rakesh@125.16.34.66) Quit (Remote host closed the connection)
[15:18] * branto (~branto@178.253.167.12) Quit (Quit: Leaving.)
[15:19] * branto (~branto@178.253.167.12) has joined #ceph
[15:25] * shaunm (~shaunm@ms-208-102-105-216.gsm.cbwireless.com) has joined #ceph
[15:29] * evelu (~erwan@2a01:e34:eecb:7400:4eeb:42ff:fedc:8ac) has joined #ceph
[15:30] * karnan (~karnan@125.16.34.66) Quit (Ping timeout: 480 seconds)
[15:31] * mhack (~mhack@nat-pool-bos-u.redhat.com) Quit (Ping timeout: 480 seconds)
[15:31] * dyasny (~dyasny@cable-192.222.152.136.electronicbox.net) has joined #ceph
[15:32] * haplo37 (~haplo37@107.190.42.94) has joined #ceph
[15:35] * mhack (~mhack@nat-pool-bos-t.redhat.com) has joined #ceph
[15:36] * bara (~bara@213.175.37.12) Quit (Ping timeout: 480 seconds)
[15:40] * salwasser (~Adium@72.246.3.14) has joined #ceph
[15:43] * boolman (boolman@79.138.78.238) has joined #ceph
[15:44] <boolman> Hey, do I need kernel 4.5 or newer if I'm going to change crush tunables from hammer to jewel?, my clients is libvirt-bin 1.2.2 running kernel 4.4
[15:45] <boolman> or is it strictly if I'm using cephfs or rbdmap I need to look at the requirements
[15:45] <boolman> http://docs.ceph.com/docs/master/rados/operations/crush-map/
[15:46] <boolman> I guess since the rbd kernel module is loaded I answered my own question
[15:47] * bara (~bara@nat-pool-brq-t.redhat.com) has joined #ceph
[15:50] * rendar (~I@95.239.179.96) has joined #ceph
[15:51] * mattbenjamin (~mbenjamin@76-206-42-50.lightspeed.livnmi.sbcglobal.net) Quit (Ping timeout: 480 seconds)
[15:51] * Skaag (~lunix@ool-ae2c3799.dyn.optonline.net) has joined #ceph
[15:52] * dugravot61 (~dugravot6@nat-persul-plg.wifi.univ-lorraine.fr) Quit (Ping timeout: 480 seconds)
[15:54] * jermudgeon (~jhaustin@199.200.6.173) has joined #ceph
[15:57] * bene2 is now known as bene_in_alu_mtg
[15:57] * bene_in_alu_mtg (~bene@nat-pool-bos-t.redhat.com) Quit (Quit: Konversation terminated!)
[15:58] * mattbenjamin (~mbenjamin@12.118.3.106) has joined #ceph
[15:59] * dugravot6 (~dugravot6@l-p-dn-in-4a.lionnois.site.univ-lorraine.fr) has joined #ceph
[16:03] * hyperbaba (~hyperbaba@private.neobee.net) Quit (Quit: Konversation terminated!)
[16:04] * salwasser (~Adium@72.246.3.14) Quit (Quit: Leaving.)
[16:04] * salwasser (~Adium@72.246.3.14) has joined #ceph
[16:05] * bearkitten (~bearkitte@cpe-76-172-86-115.socal.res.rr.com) Quit (Ping timeout: 480 seconds)
[16:10] * T1w (~jens@node3.survey-it.dk) Quit (Ping timeout: 480 seconds)
[16:11] * vata (~vata@207.96.182.162) has joined #ceph
[16:12] * dyasny (~dyasny@cable-192.222.152.136.electronicbox.net) Quit (Ping timeout: 480 seconds)
[16:18] * krypto (~krypto@G68-121-13-123.sbcis.sbc.com) has joined #ceph
[16:21] * dyasny (~dyasny@cable-192.222.152.136.electronicbox.net) has joined #ceph
[16:28] * ivve (~zed@cust-gw-11.se.zetup.net) Quit (Ping timeout: 480 seconds)
[16:28] * kefu (~kefu@114.92.125.128) has joined #ceph
[16:29] * georgem (~Adium@206.108.127.16) has joined #ceph
[16:38] * bniver (~bniver@71-9-144-29.static.oxfr.ma.charter.com) Quit (Ping timeout: 480 seconds)
[16:38] * yanzheng (~zhyan@118.116.115.254) Quit (Quit: This computer has gone to sleep)
[16:42] * salwasser (~Adium@72.246.3.14) Quit (Quit: Leaving.)
[16:42] * dyasny (~dyasny@cable-192.222.152.136.electronicbox.net) Quit (Ping timeout: 480 seconds)
[16:43] * dyasny (~dyasny@cable-192.222.152.136.electronicbox.net) has joined #ceph
[16:44] * jermudgeon (~jhaustin@199.200.6.173) Quit (Quit: jermudgeon)
[16:46] * Lokta (~Lokta@carbon.coe.int) Quit (Ping timeout: 480 seconds)
[16:50] * bniver (~bniver@71-9-144-29.static.oxfr.ma.charter.com) has joined #ceph
[17:02] * bearkitten (~bearkitte@cpe-76-172-86-115.socal.res.rr.com) has joined #ceph
[17:06] * Hemanth (~hkumar_@125.16.34.66) Quit (Ping timeout: 480 seconds)
[17:07] * schlitzer (~schlitzer@barriere.frankfurter-softwarefabrik.de) Quit (Remote host closed the connection)
[17:07] <peetaur2> for best stability, which version should I run?
[17:09] * wushudoin (~wushudoin@2601:646:8200:c9f0:2ab2:bdff:fe0b:a6ee) has joined #ceph
[17:10] * wushudoin (~wushudoin@2601:646:8200:c9f0:2ab2:bdff:fe0b:a6ee) Quit ()
[17:10] * wushudoin (~wushudoin@2601:646:8200:c9f0:2ab2:bdff:fe0b:a6ee) has joined #ceph
[17:10] * donatas (~donatas@88-119-196-104.static.zebra.lt) Quit (Quit: leaving)
[17:16] * TMM (~hp@185.5.121.201) Quit (Ping timeout: 480 seconds)
[17:18] <georgem> peetaur2: I think Hammer, but other people might think differently
[17:18] <peetaur2> oh hammer is pretty far back... you'd go that far, eh?
[17:19] <peetaur2> (and hammer is the one I found a cephfs bug where I can null parts of files)
[17:19] <peetaur2> (new files at least)
[17:19] <georgem> peetaur2: you didn't mention cephfs
[17:19] <peetaur2> do you think cephfs is a good idea? and how should I go about using that?
[17:19] <georgem> if your primary use case is cephfs, newer is better
[17:20] <peetaur2> I have 2 jobs for ceph... one is for rbd for kvm, and the other is cephfs to get rid of non-redundant NFS servers which annoy me
[17:20] <peetaur2> something trivial like updating the kernel shouldn't be such a pain
[17:20] <georgem> then use Jewel
[17:21] <peetaur2> ok and what sorts of problems would you expect in jewel that you avoid in hammer?
[17:21] * branto (~branto@178.253.167.12) Quit (Ping timeout: 480 seconds)
[17:21] * ivve (~zed@c83-254-7-248.bredband.comhem.se) has joined #ceph
[17:23] <georgem> peetaur2: radosgw used to be pretty stable in hammer, then updating to jewel broke existing buckets because it introduced new functionality which was not tested enough; every time new functionality is introduced, there is a big chance things that were working fine will break...
[17:23] <peetaur2> I have no plans to use radosgw ...don't even know what anyone uses it for other than openstack swift
[17:24] <georgem> peetaur2: if you were to use your cluster for rbd and radosgw I would recommend hammer, but if you need it cephfs then you have to go with jewel
[17:24] <peetaur2> but I know what you mean... not just radosgw, but in general, use stable instead of bleeding edge
[17:24] * salwasser (~Adium@72.246.3.14) has joined #ceph
[17:24] <peetaur2> and do you think cephfs is stable and ready to use like a big huge filesystem? (or many smaller ones...)
[17:25] <georgem> jewel is LTS, but still needs more production like testing
[17:25] <peetaur2> mostly I just want it to be possilble to do upgrades and hw fixes while the service runs
[17:26] <georgem> I'm not using cephfs and I don't think it can take a lot of IO or very large filesystems, but if you backup the data and use for smaller use cases it is probably good enough
[17:27] * karnan (~karnan@106.51.139.84) has joined #ceph
[17:27] <peetaur2> well I want to eventually replace something like 160TB of NFS storage
[17:27] <peetaur2> right now it's on 3 4U zfs machines
[17:29] <peetaur2> and I tested it a bunch with some time cp -a ... things and fio, and it seemed faster than my raid6 zfs nfs
[17:32] <peetaur2> headin' home....feel free to type and I'll continue in 30min or so at home
[17:40] * kefu (~kefu@114.92.125.128) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[17:41] <georgem> as I said, I haven't tested cephfs but follow the maling list and I've seen people with broken cephfs environments asking for help and it didn't look like it's stable at medium scale, but you'll have to test for yourself and if you're happy with it, then it's good solution for your needs
[17:45] * kefu (~kefu@114.92.125.128) has joined #ceph
[17:50] * EinstCrazy (~EinstCraz@183.195.124.35) Quit (Remote host closed the connection)
[17:54] * ntpttr (~ntpttr@134.134.139.82) has joined #ceph
[17:55] * ntpttr_ (~ntpttr@192.55.54.40) has joined #ceph
[17:55] * ntpttr_ (~ntpttr@192.55.54.40) Quit ()
[17:57] * malevolent (~quassel@192.146.172.118) Quit (Remote host closed the connection)
[17:59] * peetaur (~peter@p200300E10BC08D00667002FFFE2E10FC.dip0.t-ipconnect.de) has joined #ceph
[18:00] * bara (~bara@nat-pool-brq-t.redhat.com) Quit (Ping timeout: 480 seconds)
[18:00] * Be-El (~blinke@nat-router.computational.bio.uni-giessen.de) Quit (Quit: Leaving.)
[18:06] <peetaur> georgem: yeah joining mailing lists sounds like a good suggestion....thanks
[18:10] * borei (~dan@216.13.217.230) has joined #ceph
[18:11] * bara (~bara@213.175.37.12) has joined #ceph
[18:14] * nardial (~ls@p5DC073F9.dip0.t-ipconnect.de) has joined #ceph
[18:18] * rwheeler (~rwheeler@bzq-218-188-80.red.bezeqint.net) has joined #ceph
[18:18] * rwheeler (~rwheeler@bzq-218-188-80.red.bezeqint.net) Quit (autokilled: This host may be infected. Mail support@oftc.net with questions. BOPM (2016-09-20 16:18:18))
[18:18] * rwheeler (~rwheeler@bzq-84-111-170-30.cablep.bezeqint.net) has joined #ceph
[18:19] * kefu_ (~kefu@114.92.125.128) has joined #ceph
[18:20] * vasu (~vasu@c-73-231-60-138.hsd1.ca.comcast.net) has joined #ceph
[18:22] * salwasser1 (~Adium@72.246.3.14) has joined #ceph
[18:22] * salwasser (~Adium@72.246.3.14) Quit (Read error: Connection reset by peer)
[18:24] * xinli (~charleyst@32.97.110.53) has joined #ceph
[18:24] * kefu (~kefu@114.92.125.128) Quit (Ping timeout: 480 seconds)
[18:27] <xinli> Hi, rkeene: I hit one issue when running command ""wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add -"", no pgp valid data found. any idea to debug?
[18:28] * basiv (~oftc-webi@209-6-75-23.c3-0.abr-ubr1.sbo-abr.ma.cable.rcn.com) has joined #ceph
[18:28] <basiv> does anyone other than red hat offer support for ceph?
[18:30] * malevolent (~quassel@192.146.172.118) has joined #ceph
[18:32] * davidzlap (~Adium@2605:e000:1313:8003:60e4:8915:f702:1f2) has joined #ceph
[18:33] * vbellur (~vijay@2601:18f:700:55b0:5e51:4fff:fee8:6a5c) Quit (Ping timeout: 480 seconds)
[18:34] * TMM (~hp@dhcp-077-248-009-229.chello.nl) has joined #ceph
[18:35] * kefu_ (~kefu@114.92.125.128) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[18:37] * basiv (~oftc-webi@209-6-75-23.c3-0.abr-ubr1.sbo-abr.ma.cable.rcn.com) Quit (Quit: Page closed)
[18:38] * basiv (~oftc-webi@209-6-75-23.c3-0.abr-ubr1.sbo-abr.ma.cable.rcn.com) has joined #ceph
[18:39] * kefu (~kefu@114.92.125.128) has joined #ceph
[18:39] * kefu is now known as kefu|afk
[18:41] * bene2 (~bene@nat-pool-bos-t.redhat.com) has joined #ceph
[18:41] * Azru (~murmur@108.61.122.214) has joined #ceph
[18:45] * Skaag (~lunix@ool-ae2c3799.dyn.optonline.net) Quit (Quit: Leaving.)
[18:54] * garphy is now known as garphy`aw
[18:56] * DanFoster (~Daniel@office.34sp.com) Quit (Quit: Leaving)
[18:56] * EinstCrazy (~EinstCraz@183.195.124.35) has joined #ceph
[19:01] <peetaur> xinli: just dl it first, then view the file, then apt-key after
[19:02] <xinli> peetaur: where can I find the downloaded file in ~? the name is release.asc
[19:02] <peetaur> just like wget https://download.ceph.com/keys/release.asc
[19:03] * Nicho1as (~nicho1as@00022427.user.oftc.net) Quit (Quit: A man from the Far East; using WeeChat 1.5)
[19:04] * diver (~diver@95.85.8.93) has joined #ceph
[19:04] * doppelgrau (~doppelgra@132.252.235.172) Quit (Quit: Leaving.)
[19:04] * dgurtner (~dgurtner@84-73-130-19.dclient.hispeed.ch) Quit (Ping timeout: 480 seconds)
[19:07] <xinli> peetaur: thx
[19:11] * Azru (~murmur@2RTAAAIA4.tor-irc.dnsbl.oftc.net) Quit ()
[19:13] * xarses (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[19:23] * nardial (~ls@p5DC073F9.dip0.t-ipconnect.de) Quit (Quit: Leaving)
[19:25] * basiv (~oftc-webi@209-6-75-23.c3-0.abr-ubr1.sbo-abr.ma.cable.rcn.com) Quit (Quit: Page closed)
[19:27] * xarses (~xarses@64.124.158.3) has joined #ceph
[19:28] * Hemanth (~hkumar_@103.228.221.171) has joined #ceph
[19:32] * bara (~bara@213.175.37.12) Quit (Ping timeout: 480 seconds)
[19:37] * kefu|afk (~kefu@114.92.125.128) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[19:37] * sudocat (~dibarra@cpe-76-173-12-203.hawaii.res.rr.com) has joined #ceph
[19:38] * doppelgrau (~doppelgra@dslb-088-072-094-200.088.072.pools.vodafone-ip.de) has joined #ceph
[19:39] * vbellur (~vijay@nat-pool-bos-t.redhat.com) has joined #ceph
[19:41] * jowilkin (~jowilkin@184-23-213-254.fiber.dynamic.sonic.net) Quit (Ping timeout: 480 seconds)
[19:45] * jermudgeon (~jhaustin@gw1.ttp.biz.whitestone.link) has joined #ceph
[19:46] * Mibka (~andy@109.236.94.245) has joined #ceph
[19:50] <Mibka> I have 3 servers with 4 SSD's. Now I want to try Ceph on it. They are 512Gb in size, so the total size will be 3*4*512/2, right? About 3Tb
[19:50] <Mibka> I'm also worried about what happens if an SSD dies. For example, an SSD were the ceph journal is on too
[19:51] <T1> Mibka: by defualt ceph stores 3 copies so it should be /3
[19:52] <peetaur> Mibka: if size is 2, then yes...but default size is 3 I think (or maybe depends on how you deploy)
[19:52] <Mibka> Okay but I would like more storage, comparable to raid10. So I think i can set size=2?
[19:52] <T1> and yes - if you loose a journal SSD you should assume that all OSDs that ssd were a journal device for is also lost
[19:52] <T1> if you can live with only 2 copies
[19:52] <T1> it depends on your data
[19:53] <T1> for test-purposes it should probably be enough
[19:53] <Mibka> only 2 copies is like raid10, isn't it? sound okay to me
[19:53] <T1> but keep an eye out for "min copies"
[19:53] <T1> no it's not comparable to raid
[19:53] <peetaur> if you react instantly when there's a failure, eg. you lose a disk and are out of space and so it can't auto rebuild, ... then maybe 2 is fine for you
[19:54] <peetaur> but if you buy a batch of disks from one production run and they all are overheated because the A/C broke, etc. and 2 die at once, maybe not so good
[19:55] <T1> at cern some clusters have 5 or 7 copies because the data is so valuable (and there is so much of it that several faults at the same time is not a possibility, but a regular event)
[19:55] <peetaur> (disks from the same production run might have a common defect)
[19:55] <T1> production deployments should use at least 3
[19:56] <T1> otherwise you risk loosing data and halting all I/O to the cluster if the "right" OSDs die at the same time
[19:56] <T1> but for testing it's probably okay
[19:57] <Mibka> It's not comparable to raid you say, but if there are always two copies, this is the same as with raid10. raid10 also just has two copies of the data, right? ... So if one disk fails, no data is lost
[19:57] * mykola (~Mikolaj@91.245.73.87) has joined #ceph
[19:57] <T1> no..
[19:57] <Mibka> I mean .. other systems we have are running raid10 if two disks holding the same data die, well .. then it's dead :)
[19:57] * Racpatel (~Racpatel@2601:87:3:31e3::77ec) Quit (Ping timeout: 480 seconds)
[19:58] <T1> ceph is not comparable to raid in any way
[19:58] <peetaur> I'd say it achieves a similar goal in a very different way
[19:58] <Mibka> okay
[19:59] <Mibka> but ceph with size=2 is equally safe as raid10? in terms of what happens if multiple disks die
[20:00] <diver> depends on the policy. with ruleset-failure-domain=host if 2 disks dies on the same server then you're still safe
[20:00] <peetaur> as said earlier, if they have separate but shared journals, then it's more than 1 disk dying when the journal dies
[20:00] <peetaur> plus it auto rebuilds like a spare (unless out of space)
[20:00] <diver> with ruleset-failure-domain=osd it could be chance that both failed OSD's stores same objects (size=2)
[20:01] * xinli (~charleyst@32.97.110.53) Quit (Ping timeout: 480 seconds)
[20:01] * georgem (~Adium@206.108.127.16) Quit (Read error: Connection reset by peer)
[20:01] <T1> so.. do not think of it as raid.. it's a flawed comparison
[20:01] * georgem (~Adium@206.108.127.16) has joined #ceph
[20:01] <T1> it's like apples and oranges
[20:01] <johnavp1989> I don't think it's accurate to assume a shared journal = all OSD's using it are dead. You can replace the journal the the OSD's will recover. Though they are offline until the journal is replaced.
[20:02] <T1> johnavp1989: there are ways to recover data, but you should not assume you can
[20:03] <T1> you can easily risk some data only to be on the journal and not yet flushed to the OSD data drive
[20:03] <johnavp1989> T1: Yes but that data should exist on another journal
[20:04] <T1> if it's the primary copy then the copy from other OSDs are not valid
[20:04] <peetaur> if a journal dies, but the machine is online, shouldn't the data be in ram still, so the osds are safe unless it loses power, panics, etc.?
[20:04] <T1> unless you discard the entire placement group
[20:04] <peetaur> I see the journal getting writes and never read, so that must be true
[20:05] <T1> peetaur: data is written to the journal and then flushed to the data disk at set intervals
[20:05] * swami1 (~swami@27.7.172.83) has joined #ceph
[20:05] <peetaur> but is it flushed from RAM or from the journal? I see no read on the journal so it is in RAM...either linux block cache, or in some buffer still being written
[20:05] <T1> from the journal
[20:06] <T1> you might have the OS cache in play, but it depends on how busy your cluster is - it could as easily have been read from the journal
[20:06] <peetaur> maybe I"ll try a vm.drop_caches=3 during a write to see if it reads
[20:06] <johnavp1989> T1: Wat do you mean the primary copy? If your replication is set to 3 all data should be written to 3 different journals before being flushed to disk unless your policies are not configure intelligently right?
[20:07] <T1> at least that what the documentation says..
[20:09] * ivve (~zed@c83-254-7-248.bredband.comhem.se) Quit (Ping timeout: 480 seconds)
[20:09] <T1> johnavp1989: there are primary PGs in ceph that data is read from
[20:12] <T1> you could of course promote the other copies to primary and get data backfilled from them
[20:12] <Mibka> okay. and what exactly is the difference between ruleset-failure-domain=osd and ruleset-failure-domain=host ? With host it's allowed to lose an entire host?
[20:12] <johnavp1989> T1: okay so does the same thing apply when I loose a primary PG that has already been flushed to disk?
[20:12] * sudocat (~dibarra@cpe-76-173-12-203.hawaii.res.rr.com) Quit (Ping timeout: 480 seconds)
[20:15] <T1> Mibka: a failure domain is a way of ensuring certain failures are covered
[20:15] <T1> johnavp1989: afaik, yes
[20:15] * stiopa (~stiopa@cpc73832-dals21-2-0-cust453.20-2.cable.virginm.net) has joined #ceph
[20:16] <peetaur> Mibka: failure domain is to balance between performance of replication and resiliency to failure (pci bus/controller vs network hardware)
[20:16] <peetaur> (or internal network, vs. WAN connection through datacenters, etc.)
[20:17] * sphinxx (~sphinxx@154.118.120.40) has joined #ceph
[20:17] * swami1 (~swami@27.7.172.83) Quit (Quit: Leaving.)
[20:21] * donatas (~donatas@78.63.213.199) has joined #ceph
[20:22] * fdmanana (~fdmanana@2001:8a0:6e0c:6601:35fa:21a0:3929:f94d) Quit (Ping timeout: 480 seconds)
[20:24] <Mibka> quit bbl
[20:24] * Mibka (~andy@109.236.94.245) Quit (Quit: bbl)
[20:27] * karnan (~karnan@106.51.139.84) Quit (Ping timeout: 480 seconds)
[20:28] * mnathani (~mnathani@192-0-149-228.cpe.teksavvy.com) Quit ()
[20:29] * ledgr_ (~ledgr@88-222-11-185.meganet.lt) has joined #ceph
[20:29] * jowilkin (~jowilkin@172.56.39.108) has joined #ceph
[20:29] * TomasCZ (~TomasCZ@yes.tenlab.net) has joined #ceph
[20:30] * ledgr_ (~ledgr@88-222-11-185.meganet.lt) Quit (Remote host closed the connection)
[20:30] <peetaur> "mon.ceph1 low disk space" what's the limit for that warning? is it configurable?
[20:30] * georgem (~Adium@206.108.127.16) Quit (Read error: Connection reset by peer)
[20:30] * ledgr_ (~ledgr@88-222-11-185.meganet.lt) has joined #ceph
[20:30] * georgem (~Adium@206.108.127.16) has joined #ceph
[20:31] <peetaur> (just a test cluster on my desktop :) with 25G/32G 83% used)
[20:34] * xinli (~charleyst@32.97.110.50) has joined #ceph
[20:34] * tmartiro (~tmartiro@82.199.196.207) has joined #ceph
[20:35] <tmartiro> hi folks, could you please help me with creating public bucket where everyone has access for a read using radosgw-admin?
[20:36] * ledgr (~ledgr@88-119-196-104.static.zebra.lt) Quit (Ping timeout: 480 seconds)
[20:36] <tmartiro> I cannot figure out how can I set caps
[20:37] <tmartiro> is anybody here ?
[20:38] * ledgr_ (~ledgr@88-222-11-185.meganet.lt) Quit (Ping timeout: 480 seconds)
[20:39] * karnan (~karnan@2405:204:540d:41a2:3602:86ff:fe56:55ae) has joined #ceph
[20:43] * tmartiro_ (~tmartiro@82.199.196.207) has joined #ceph
[20:44] <tmartiro_> hi folks, could you please help me with creating public bucket where everyone has access for a read using radosgw-admin?
[20:44] * tmartiro_ (~tmartiro@82.199.196.207) Quit ()
[20:45] <peetaur> I would if I knew anything about rados :)
[20:45] <peetaur> radosgw that is
[20:45] * rakeshgm (~rakesh@106.51.29.253) has joined #ceph
[20:45] <tmartiro> peetaur: 10x anyway
[20:47] <donatas> guys, what's your opinion about RBD+NFS?
[20:54] <diver> tmartiro: as far as as I see from radosgw-admin tool it doesn't support bucket creation. I would suggest you to use s3cmd instead
[20:54] <tmartiro> I used it , but I am not able to create policy for the bucket
[20:55] <tmartiro> s3cmd setpolicy media.policy s3://media/
[20:55] <tmartiro> does not work for me
[20:55] <tmartiro> https://gist.github.com/anonymous/8ca0c4d8fab5dd5d49437e4d1221ed68
[20:56] <tmartiro> here is the media.policy file content looks like
[20:56] <diver> check this: https://stackoverflow.com/questions/31853383/riak-cs-public-acl-by-default-for-new-objects
[20:57] <tmartiro> same error
[20:57] <tmartiro> ERROR: S3 error: 400 (InvalidArgument)
[20:58] <tmartiro> I am starting to think that radosgw is not fully compatible with S3
[20:59] <diver> then use
[20:59] <diver> s3cmd mb s3://test
[20:59] <diver> s3cmd setacl --acl-public s3://test
[21:00] <diver> "s3://test/: ACL set to Public "
[21:00] <diver> then it's accessible
[21:00] <diver> just checked
[21:00] * jarrpa (~jarrpa@4.28.11.153) has joined #ceph
[21:00] <diver> all the s3 clients I checke
[21:00] <diver> all the s3 clients I checked*
[21:00] <diver> works perfectly with radosgw
[21:01] <tmartiro> ok, now put file in test
[21:01] <tmartiro> and then access from outside
[21:02] <tmartiro> when I invoke the command s3cmd setacl --acl-public s3://media
[21:03] <tmartiro> the output is nothing
[21:04] <diver> then it depends on the put request policy
[21:04] <diver> s3cmd put --acl-public /etc/hosts s3://test
[21:04] <diver> upload: '/etc/hosts' -> 's3://test/hosts' [1 of 1]
[21:04] <diver> 196 of 196 100% in 0s 1804.29 B/s done
[21:04] <diver> Public URL of the object is: http://test.host/test/hosts
[21:04] <diver> that's the output
[21:04] <diver> and it's accessible
[21:05] <tmartiro> do you configure radosgw with apache2.4 + mod_proxy
[21:05] * krypto (~krypto@G68-121-13-123.sbcis.sbc.com) Quit (Ping timeout: 480 seconds)
[21:05] <diver> no... it's citweb
[21:05] <tmartiro> *mod_proxy_fcgi
[21:05] <diver> with Jewel
[21:05] * malevolent (~quassel@192.146.172.118) Quit (Quit: No Ping reply in 180 seconds.)
[21:06] <tmartiro> ok thanks
[21:06] <diver> # ps auxww|grep rgw
[21:06] <diver> ceph 3134399 71.9 3.4 70215964 415848 ? Ssl 12:33 109:42 /usr/bin/radosgw -f --cluster ceph --name client.rgw.ceph-mon03 --setuser ceph --setgroup ceph
[21:07] * malevolent (~quassel@192.146.172.118) has joined #ceph
[21:07] * derjohn_mob (~aj@46.189.28.53) Quit (Ping timeout: 480 seconds)
[21:09] <tmartiro> I am on ubuntu. Is there any chance to run citweb on it ?
[21:09] * donatas (~donatas@78.63.213.199) Quit (Quit: leaving)
[21:10] <diver> why not. just install it with ceph-deploy
[21:11] * Eman1 (~dux0r@exit0.radia.tor-relays.net) has joined #ceph
[21:11] * rakeshgm (~rakesh@106.51.29.253) Quit (Quit: Peace :))
[21:12] * garphy`aw is now known as garphy
[21:12] * salwasser1 (~Adium@72.246.3.14) Quit (Quit: Leaving.)
[21:14] * diver_ (~diver@216.85.162.34) has joined #ceph
[21:14] <tmartiro> diver: can you point to any documentation related citweb. I found nothing
[21:14] * peetaur is now known as Guest1013
[21:14] * peetaur (~peter@p57AAAF92.dip0.t-ipconnect.de) has joined #ceph
[21:15] <peetaur> oops....broke my test osd somehow and then mds died and had to kill it, and then the computer froze. :/
[21:15] <peetaur> long ago you couldn't run a mon and mount cephfs on the same machine... is that still true? (this was jewel 10.2.2)
[21:17] <peetaur> computer froze after starting the osd then killing and starting the mds
[21:19] <diver_> ceph-deploy --overwrite-conf install --rgw ceph-test
[21:19] <diver_> ceph-deploy --overwrite-conf rgw create ceph-test
[21:19] <diver_> tmartiro:
[21:19] <diver_> that's how I installed that manually
[21:19] <diver_> where ceph-test is the server hostname
[21:20] * Guest1013 (~peter@p200300E10BC08D00667002FFFE2E10FC.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[21:20] <diver_> tmartiro: http://docs.ceph.com/ceph-deploy/docs/rgw.html
[21:20] * diver (~diver@95.85.8.93) Quit (Ping timeout: 480 seconds)
[21:21] <tmartiro> haha, it worked with my fastcgi installation, s3cmd setacl --acl-public s3://media/*
[21:21] <diver_> good!
[21:21] <tmartiro> does it mean that everybody can also write there ?
[21:21] * rendar (~I@95.239.179.96) Quit (Ping timeout: 480 seconds)
[21:22] <diver_> no
[21:22] <diver_> -P, --acl-public Store objects with ACL allowing read for anyone.
[21:22] <diver_> --acl-grant will give write perms
[21:23] <tmartiro> diver_: thank you very much. I love you man )))
[21:23] * bene2 (~bene@nat-pool-bos-t.redhat.com) Quit (Quit: Konversation terminated!)
[21:25] * krypto (~krypto@G68-121-13-123.sbcis.sbc.com) has joined #ceph
[21:25] * krypto (~krypto@G68-121-13-123.sbcis.sbc.com) Quit ()
[21:29] * Racpatel (~Racpatel@2601:87:3:31e3::77ec) has joined #ceph
[21:30] * diver_ (~diver@216.85.162.34) Quit (Remote host closed the connection)
[21:30] * Racpatel (~Racpatel@2601:87:3:31e3::77ec) Quit ()
[21:30] * diver (~diver@216.85.162.38) has joined #ceph
[21:31] <diver> sure
[21:39] * evelu (~erwan@2a01:e34:eecb:7400:4eeb:42ff:fedc:8ac) Quit (Ping timeout: 480 seconds)
[21:41] * Eman1 (~dux0r@635AAAP1A.tor-irc.dnsbl.oftc.net) Quit ()
[21:48] * rendar (~I@95.239.179.96) has joined #ceph
[21:49] <peetaur> osd died again... ceph -s says "osdmap e334: 1 osds: 1 up, 1 in" and cephfs is hung again.
[21:49] <peetaur> (9 minutes passed, still doesn't say osd is down)
[21:56] * EinstCrazy (~EinstCraz@183.195.124.35) Quit (Remote host closed the connection)
[21:56] * karnan (~karnan@2405:204:540d:41a2:3602:86ff:fe56:55ae) Quit (Ping timeout: 480 seconds)
[21:58] <peetaur> finally it says the osd is down
[22:04] * jowilkin (~jowilkin@172.56.39.108) Quit (Ping timeout: 480 seconds)
[22:10] * karnan (~karnan@106.51.139.84) has joined #ceph
[22:11] * georgem (~Adium@206.108.127.16) Quit (Quit: Leaving.)
[22:13] * jarrpa (~jarrpa@4.28.11.153) Quit (Ping timeout: 480 seconds)
[22:17] * bniver (~bniver@71-9-144-29.static.oxfr.ma.charter.com) Quit (Remote host closed the connection)
[22:18] * madkiss (~madkiss@2a02:8109:8680:2000:95b7:2e70:7e03:1e22) Quit (Quit: Leaving.)
[22:19] * madkiss (~madkiss@vpn143.sys11.net) has joined #ceph
[22:19] * bene2 (~bene@nat-pool-bos-t.redhat.com) has joined #ceph
[22:20] * LegalResale (~LegalResa@66.165.126.130) has joined #ceph
[22:22] * bene2 is now known as bene_in_alu_mtg
[22:22] * bene_in_alu_mtg is now known as bene2
[22:24] * mykola (~Mikolaj@91.245.73.87) Quit (Quit: away)
[22:28] * wkennington (~wkenningt@c-71-204-170-241.hsd1.ca.comcast.net) has joined #ceph
[22:33] * garphy is now known as garphy`aw
[22:33] * diver_ (~diver@216.85.162.34) has joined #ceph
[22:38] * xinli (~charleyst@32.97.110.50) Quit (Ping timeout: 480 seconds)
[22:39] * Jeffrey4l__ (~Jeffrey@110.252.65.44) Quit (Ping timeout: 480 seconds)
[22:41] * diver (~diver@216.85.162.38) Quit (Ping timeout: 480 seconds)
[22:41] * diver_ (~diver@216.85.162.34) Quit (Ping timeout: 480 seconds)
[22:43] * Hemanth (~hkumar_@103.228.221.171) Quit (Ping timeout: 480 seconds)
[22:48] * tmartiro (~tmartiro@82.199.196.207) Quit (Remote host closed the connection)
[22:49] * xinli (~charleyst@32.97.110.56) has joined #ceph
[22:50] * madkiss1 (~madkiss@2a02:8109:8680:2000:617f:5275:ba6d:2b5a) has joined #ceph
[22:50] * rraja (~rraja@125.16.34.66) Quit (Quit: Leaving)
[22:56] * madkiss (~madkiss@vpn143.sys11.net) Quit (Ping timeout: 480 seconds)
[23:05] * ledgr (~ledgr@88-222-11-185.meganet.lt) has joined #ceph
[23:06] * Solvius (~adept256@tor2r.ins.tor.net.eu.org) has joined #ceph
[23:09] * karnan (~karnan@106.51.139.84) Quit (Ping timeout: 480 seconds)
[23:15] * georgem (~Adium@24.114.71.69) has joined #ceph
[23:15] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[23:20] * jermudgeon (~jhaustin@gw1.ttp.biz.whitestone.link) Quit (Quit: jermudgeon)
[23:27] <peetaur> oh I guess it takes long for my single osd to be reported down because "... a ceph-osd process dies, the monitor will learn about the failure **from surviving ceph-osd daemons** and report it via the ceph health command"
[23:29] * mhack (~mhack@nat-pool-bos-t.redhat.com) Quit (Remote host closed the connection)
[23:34] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[23:34] * georgem (~Adium@24.114.71.69) Quit (Quit: Leaving.)
[23:35] * Solvius (~adept256@5AEAABSDK.tor-irc.dnsbl.oftc.net) Quit ()
[23:37] * Anvilou (~anvil@2a01:c9c0:a1:1000:0:aff:fe6b:ff) has joined #ceph
[23:39] * anvil (~anvil@2a01:c9c0:a1:1000:0:aff:fe6b:ff) Quit (Ping timeout: 480 seconds)
[23:40] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[23:41] * Racpatel (~Racpatel@2601:87:3:31e3::77ec) has joined #ceph
[23:45] * vbellur (~vijay@nat-pool-bos-t.redhat.com) Quit (Quit: Leaving.)
[23:46] * rendar (~I@95.239.179.96) Quit (Quit: std::lower_bound + std::less_equal *works* with a vector without duplicates!)
[23:52] * jermudgeon (~jhaustin@gw1.ttp.biz.whitestone.link) has joined #ceph
[23:52] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[23:55] * jowilkin (~jowilkin@184-23-213-254.fiber.dynamic.sonic.net) has joined #ceph
[23:55] * mattbenjamin (~mbenjamin@12.118.3.106) Quit (Ping timeout: 480 seconds)
[23:58] * ledgr (~ledgr@88-222-11-185.meganet.lt) Quit (Remote host closed the connection)
[23:58] * johnavp1989 (~jpetrini@8.39.115.8) Quit (Ping timeout: 480 seconds)
[23:58] * ledgr (~ledgr@88-119-196-104.static.zebra.lt) has joined #ceph

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.