#ceph IRC Log

Index

IRC Log for 2016-04-11

Timestamps are in GMT/BST.

[0:06] * vicente (~vicente@111-241-31-95.dynamic.hinet.net) has joined #ceph
[0:06] * DV (~veillard@2001:41d0:a:f29f::1) Quit (Remote host closed the connection)
[0:07] * DV (~veillard@2001:41d0:a:f29f::1) has joined #ceph
[0:14] * vicente (~vicente@111-241-31-95.dynamic.hinet.net) Quit (Ping timeout: 480 seconds)
[0:15] * Azru (~Pommesgab@76GAAEBQY.tor-irc.dnsbl.oftc.net) Quit ()
[0:20] * Arcturus1 (~rogst@76GAAEBSC.tor-irc.dnsbl.oftc.net) has joined #ceph
[0:24] * dgurtner (~dgurtner@89.248.139.44) has joined #ceph
[0:32] * Lea (~LeaChim@host86-159-239-193.range86-159.btcentralplus.com) Quit (Ping timeout: 480 seconds)
[0:35] * aj__ (~aj@x590e27eb.dyn.telefonica.de) has joined #ceph
[0:38] * avozza (~textual@a83-162-204-36.adsl.xs4all.nl) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[0:48] * stein (~stein@185.56.185.82) Quit (Ping timeout: 480 seconds)
[0:49] * Arcturus1 (~rogst@76GAAEBSC.tor-irc.dnsbl.oftc.net) Quit ()
[0:50] * brianjjo (~neobenedi@65.19.167.132) has joined #ceph
[0:51] * rendar (~I@host135-179-dynamic.6-87-r.retail.telecomitalia.it) Quit (Quit: std::lower_bound + std::less_equal *works* with a vector without duplicates!)
[0:56] * DV (~veillard@2001:41d0:a:f29f::1) Quit (Remote host closed the connection)
[0:57] * stein (~stein@185.56.185.82) has joined #ceph
[1:00] * chasmo77 (~chas77@158.183-62-69.ftth.swbr.surewest.net) has joined #ceph
[1:01] * DV (~veillard@2001:41d0:a:f29f::1) has joined #ceph
[1:01] * DV (~veillard@2001:41d0:a:f29f::1) Quit (Remote host closed the connection)
[1:04] * bjornar__ (~bjornar@ti0099a430-1561.bb.online.no) Quit (Ping timeout: 480 seconds)
[1:07] * vicente (~vicente@111-241-31-95.dynamic.hinet.net) has joined #ceph
[1:08] * dgurtner (~dgurtner@89.248.139.44) Quit (Ping timeout: 480 seconds)
[1:10] * DV (~veillard@2001:41d0:a:f29f::1) has joined #ceph
[1:15] * vicente (~vicente@111-241-31-95.dynamic.hinet.net) Quit (Ping timeout: 480 seconds)
[1:19] * brianjjo (~neobenedi@4MJAAD41Y.tor-irc.dnsbl.oftc.net) Quit ()
[1:20] * Arfed (~phyphor@06SAAA5KF.tor-irc.dnsbl.oftc.net) has joined #ceph
[1:26] * Eduardo_ (~Eduardo@85.193.28.37.rev.vodafone.pt) has joined #ceph
[1:28] * TomasCZ (~TomasCZ@yes.tenlab.net) Quit (Ping timeout: 480 seconds)
[1:49] * oms101 (~oms101@p20030057EA3FDA00C6D987FFFE4339A1.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[1:49] * Arfed (~phyphor@06SAAA5KF.tor-irc.dnsbl.oftc.net) Quit ()
[1:54] * Popz (~nartholli@109.201.133.100) has joined #ceph
[1:58] * oms101 (~oms101@p20030057EA029900C6D987FFFE4339A1.dip0.t-ipconnect.de) has joined #ceph
[2:00] * ivancich (~ivancich@aa2.linuxbox.com) has joined #ceph
[2:01] * ivancich_ (~ivancich@aa2.linuxbox.com) Quit (Read error: Connection reset by peer)
[2:07] * vicente (~vicente@111-241-31-95.dynamic.hinet.net) has joined #ceph
[2:11] * ivancich (~ivancich@aa2.linuxbox.com) Quit (Ping timeout: 480 seconds)
[2:15] * vicente (~vicente@111-241-31-95.dynamic.hinet.net) Quit (Ping timeout: 480 seconds)
[2:21] * vZerberus (~dogtail@00021993.user.oftc.net) has joined #ceph
[2:24] * Popz (~nartholli@4MJAAD42T.tor-irc.dnsbl.oftc.net) Quit ()
[2:24] * jwandborg (~Bwana@ns330209.ip-5-196-66.eu) has joined #ceph
[2:28] * ivancich (~ivancich@aa2.linuxbox.com) has joined #ceph
[2:29] * huangjun (~kvirc@113.57.168.154) has joined #ceph
[2:36] <TMM> I have one osd in my logs that every 15 minutes exactly shows this in clients: libceph: osd163 10.66.0.16:6802 socket closed (con state OPEN)
[2:36] <TMM> it's always the same one, and it's always exactly 15 minutes between these messages
[2:38] <TMM> it's also the only osd that seems to do any logging, interesting
[2:47] * jdillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) Quit (Read error: Connection reset by peer)
[2:47] <huangjun> if you client write/read data to/from that osd, and the connection idel 900s with no data transmission, the connection will be stopped by osd or client, so you can get this message
[2:47] <TMM> oh, so I shouldn't worry about it then
[2:48] <TMM> these are juts some scratch volumes that will most of the time see zero reads or writes
[2:49] <TMM> huangjun, thank you!
[2:49] <huangjun> yes, it's a normal output
[2:52] <TMM> ok, the only thing that that osd appears to be logging is messages about going to sleep
[2:52] <TMM> so that makes sense too then
[2:52] * vicente (~vicente@111-241-31-95.dynamic.hinet.net) has joined #ceph
[2:54] * jwandborg (~Bwana@76GAAEBW9.tor-irc.dnsbl.oftc.net) Quit ()
[2:54] * Sophie1 (~hyst@192.42.115.101) has joined #ceph
[3:03] * Eduardo__ (~Eduardo@85.193.28.37.rev.vodafone.pt) has joined #ceph
[3:05] <Eduardo__> Hello everyone, need some help. I'm trying to run 'ceph-deploy mon create-initial' but I keep getting:
[3:06] <Eduardo__> [monitor][ERROR ] admin_socket: exception getting command description: [Errno 2] No such file or directory
[3:08] * derjohn_mobi (~aj@x590cd707.dyn.telefonica.de) has joined #ceph
[3:09] <Eduardo__> Before that it mentions /var/run/ceph/ceph-mon.monitor.asok so I believe it is reffering to that. On the monitor node I tried to access that file, but it doesn't allow me to go further that /var/run, if I try to open /var/run/ceph it tells me 'Permission denied' even with 'sudo'
[3:09] * vicente (~vicente@111-241-31-95.dynamic.hinet.net) Quit (Ping timeout: 480 seconds)
[3:10] <Eduardo__> Anyway, I'm at a loss here. Thanks in advance for any help.
[3:11] * EinstCrazy (~EinstCraz@58.247.119.250) has joined #ceph
[3:16] * aj__ (~aj@x590e27eb.dyn.telefonica.de) Quit (Ping timeout: 480 seconds)
[3:19] * atheism (~atheism@182.48.117.114) has joined #ceph
[3:24] * Sophie1 (~hyst@06SAAA5LR.tor-irc.dnsbl.oftc.net) Quit ()
[3:24] * Izanagi (~Lite@freedom.ip-eend.nl) has joined #ceph
[3:35] * EinstCra_ (~EinstCraz@58.247.119.250) has joined #ceph
[3:35] * EinstCrazy (~EinstCraz@58.247.119.250) Quit (Read error: Connection reset by peer)
[3:41] <darkfader> Eduardo__: it sounds like something is mounted over your /var/run
[3:42] <darkfader> can you try doing df /var/run and see if there's something weird with that
[3:42] <darkfader> just guessing but it doesn't look like a ceph issue at all
[3:46] <Eduardo__> darkfader: thak you, it says only it is mounted on tmpfs
[3:47] * igi (~igi@218.107.55.253) has joined #ceph
[3:50] * Skaag (~lunix@65.200.54.234) Quit (Quit: Leaving.)
[3:51] * IvanJobs (~hardes@103.50.11.146) has joined #ceph
[3:54] * Izanagi (~Lite@76GAAEBZ6.tor-irc.dnsbl.oftc.net) Quit ()
[3:54] * KrimZon (~pepzi@freedom.ip-eend.nl) has joined #ceph
[3:56] <IvanJobs> hi, experts here.
[3:57] <motk> sup
[3:58] <IvanJobs> I have a questions about maintainace of ceph. I used ceph-deploy to deploy ceph hammer. When I did the deployment, every node of ceph cluster should have to modify /etc/hosts to give hostname info of all these cluster.
[3:59] <IvanJobs> I think it sucks. So Can I make a local DNS server to do the hostname resolving? just to confirm it:)
[3:59] <IvanJobs> or the best practise of it is "DNS", right?
[4:00] <motk> Not sure why it does that
[4:00] <motk> but yes, dns is important
[4:00] <Nats_> you can use dns or /etc/hosts, yes
[4:02] <Nats_> fwiw, at least in practice, you dont need every node to know the name of everything else
[4:02] * vicente (~vicente@111-241-31-95.dynamic.hinet.net) has joined #ceph
[4:03] <IvanJobs> well, maybe I'm wrong. When deploying ceph, we don't have to modify nodes of ceph cluster, we just need to info ceph-admin node all of the nodes in ceph cluster?
[4:03] <IvanJobs> Am I right?
[4:03] <Eduardo__> I found something from someone getting the same output as I am: "Your most recent error is indicative of the ceph-mon service not coming up successfully. when ceph-mon (the service, not the daemon) is started, it also calls ceph-create-keys, which waits for the monitor daemon to come up and then creates keys that are necessary for all clusters to run when using cephx (the admin key, the bootsraps keys)."
[4:03] <Eduardo__> what should be required to launch ceph-mon correctly?
[4:04] <Nats_> IvanJobs, well i'll give you a basic example - i used /etc/hosts. when i add a new host with 8 OSDs, i do not update my existing monitors or OSD hosts to know the name of the new one
[4:04] <Nats_> IvanJobs, I have to update /etc/hosts on the server I run ceph-deploy on so that can work
[4:05] <Nats_> but anecdotally at least, nothing else cares about OSD host names
[4:05] <IvanJobs> Nats_, thx I will try that a few moments later.
[4:05] <IvanJobs> I want to add a OSD to my ceph cluster this morning.
[4:05] <Nats_> IvanJobs, i have a prebuilt /etc/hosts i deploy on new osd hosts that already has the monitor hostnames, but thats it
[4:07] <IvanJobs> Nats_, do you mean that you need to add /etc/hosts for the new node of OSD to make it know all of hostnames in the cluster right now?
[4:08] <Nats_> IvanJobs, the new node gets an /etc/hosts that has my 3 monitors
[4:09] <IvanJobs> <Eduardo__> ,I have disabled cephx in my cluster since I'm in a private network safe enough, maybe not related to your question, just FYI.
[4:09] <IvanJobs> Nats_, yep, just need to add hostname resolving of ceph-mon nodes.
[4:10] * EinstCra_ (~EinstCraz@58.247.119.250) Quit (Remote host closed the connection)
[4:10] * vicente (~vicente@111-241-31-95.dynamic.hinet.net) Quit (Ping timeout: 480 seconds)
[4:13] <Eduardo__> IvanJobs: I'm also testing this on a private network. I guess I should disable it too. It is not part of my job to worry about security at this point. Let me see if that helps
[4:14] <IvanJobs> Nats_, every node of ceph cluster has a /etc/ceph/ceph.conf file, in this file, we can get ceph mon nodes' IP, so maybe we don't need to modify /etc/hosts, I just wondering which part of these things need hostname resolving?
[4:15] * yanzheng (~zhyan@125.70.21.212) has joined #ceph
[4:15] * EinstCrazy (~EinstCraz@58.247.119.250) has joined #ceph
[4:20] <Nats_> IvanJobs, no idea sorry. you are right, the ip's are listed there. the way ceph uses hostnames is certainly not intuitive
[4:22] <IvanJobs> Nats_, yep, I will try it later. Maybe we just need to add hostname resolving in ceph-admin node for convinence, we don't need to type long ip for cluster nodes.
[4:24] * KrimZon (~pepzi@4MJAAD44N.tor-irc.dnsbl.oftc.net) Quit ()
[4:24] * Spikey (~FierceFor@91.213.8.235) has joined #ceph
[4:28] * vbellur (~vijay@71.234.224.255) has joined #ceph
[4:34] <Eduardo__> IvanJobs, didn't do much difference, mon is still not launching :(
[4:35] <tacodog40k> does anyone here use ceph as a datastore in vmware, with iscsi
[4:47] * _28_ria (~kvirc@opfr028.ru) Quit (Read error: Connection reset by peer)
[4:47] * kefu (~kefu@183.193.127.235) has joined #ceph
[4:49] * zhaochao (~zhaochao@125.39.9.144) has joined #ceph
[4:50] * igi_ (~igi@218.107.55.254) has joined #ceph
[4:54] * Spikey (~FierceFor@6AGAAAPXN.tor-irc.dnsbl.oftc.net) Quit ()
[4:54] * nastidon (~Kyso_@hessel2.torservers.net) has joined #ceph
[4:55] * _28_ria (~kvirc@opfr028.ru) has joined #ceph
[4:55] * atheism (~atheism@182.48.117.114) Quit (Ping timeout: 480 seconds)
[4:57] * igi (~igi@218.107.55.253) Quit (Ping timeout: 480 seconds)
[4:57] * atheism (~atheism@182.48.117.114) has joined #ceph
[5:01] <jidar> that seems like the long way around
[5:04] * bjornar__ (~bjornar@ti0099a430-1561.bb.online.no) has joined #ceph
[5:05] * _28_ria (~kvirc@opfr028.ru) Quit (Read error: Connection reset by peer)
[5:06] * kefu_ (~kefu@114.92.120.83) has joined #ceph
[5:07] * atheism (~atheism@182.48.117.114) Quit (Read error: Connection reset by peer)
[5:11] * kefu (~kefu@183.193.127.235) Quit (Ping timeout: 480 seconds)
[5:19] * DV (~veillard@2001:41d0:a:f29f::1) Quit (Remote host closed the connection)
[5:19] * DV (~veillard@2001:41d0:a:f29f::1) has joined #ceph
[5:21] * Vacuum__ (~Vacuum@88.130.220.135) has joined #ceph
[5:24] * nastidon (~Kyso_@76GAAEB3K.tor-irc.dnsbl.oftc.net) Quit ()
[5:24] * CorneliousJD|AtWork (~Salamande@ns381528.ip-94-23-247.eu) has joined #ceph
[5:27] * Vacuum_ (~Vacuum@88.130.222.61) Quit (Ping timeout: 480 seconds)
[5:54] * CorneliousJD|AtWork (~Salamande@06SAAA5N8.tor-irc.dnsbl.oftc.net) Quit ()
[5:54] * redbeast12 (~Phase@exit1.ipredator.se) has joined #ceph
[5:57] * bjornar__ (~bjornar@ti0099a430-1561.bb.online.no) Quit (Ping timeout: 480 seconds)
[6:00] <Eduardo__> Question: ceph-deploy shuldnt add initial monitors authomatically to monmap?
[6:00] <Eduardo__> and generate monmap?
[6:01] * haplo37 (~haplo37@107.190.32.70) has joined #ceph
[6:03] * bjornar__ (~bjornar@ti0099a430-1561.bb.online.no) has joined #ceph
[6:18] * vikhyat (~vumrao@121.244.87.116) has joined #ceph
[6:20] * kefu_ is now known as kefu|afk
[6:22] <Eduardo_> ok, meanwhile
[6:23] <Eduardo_> [ceph1][WARNIN] provided hostname must match remote hostname
[6:23] <Eduardo_> [ceph1][WARNIN] provided hostname: ceph1
[6:23] <Eduardo_> [ceph1][WARNIN] remote hostname: localhost
[6:24] <Eduardo_> where is this localhost set?
[6:24] <Eduardo_> on the monitors hosts file?
[6:24] * redbeast12 (~Phase@76GAAEB5D.tor-irc.dnsbl.oftc.net) Quit ()
[6:24] * eXeler0n (~sese_@192.42.115.101) has joined #ceph
[6:26] * bjornar__ (~bjornar@ti0099a430-1561.bb.online.no) Quit (Ping timeout: 480 seconds)
[6:28] * swami1 (~swami@49.32.0.102) has joined #ceph
[6:34] * chengmao (~chengmao@113.57.168.154) Quit (Remote host closed the connection)
[6:35] * RameshN (~rnachimu@121.244.87.117) has joined #ceph
[6:48] * derjohn_mobi (~aj@x590cd707.dyn.telefonica.de) Quit (Ping timeout: 480 seconds)
[6:49] * igi_ (~igi@218.107.55.254) Quit (Quit: Leaving...)
[6:52] * wjw-freebsd (~wjw@smtp.digiware.nl) Quit (Ping timeout: 480 seconds)
[6:54] * eXeler0n (~sese_@06SAAA5OV.tor-irc.dnsbl.oftc.net) Quit ()
[6:54] * Jyron (~Moriarty@195.22.126.119) has joined #ceph
[6:57] * EinstCrazy (~EinstCraz@58.247.119.250) Quit (Remote host closed the connection)
[6:57] * Concubidated (~cube@c-50-173-245-118.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[7:01] * EinstCrazy (~EinstCraz@58.247.117.134) has joined #ceph
[7:03] * huangjun|2 (~kvirc@113.57.168.154) has joined #ceph
[7:03] * overclk (~quassel@121.244.87.117) has joined #ceph
[7:05] * _28_ria (~kvirc@opfr028.ru) has joined #ceph
[7:07] * Concubidated (~cube@c-50-173-245-118.hsd1.ca.comcast.net) has joined #ceph
[7:09] * dyasny (~dyasny@46-117-8-108.bb.netvision.net.il) has joined #ceph
[7:09] <kefu|afk> Eduardo_, it's /etc/hostname i guess?
[7:09] * kefu|afk is now known as kefu_
[7:09] * huangjun (~kvirc@113.57.168.154) Quit (Ping timeout: 480 seconds)
[7:09] * kefu_ is now known as kefu
[7:11] <Eduardo_> kefu|afk, my first guess as well, ended up discovering that I needed asystem administration tool, hostnamectl worked
[7:11] <Eduardo_> Also think I found theissue with me
[7:12] <Eduardo_> not enough RAM for the monitor
[7:12] <kefu> Eduardo_, good to learn that you root caused it =)
[7:13] * karnan (~karnan@121.244.87.117) has joined #ceph
[7:15] * bearkitten (~bearkitte@cpe-76-172-86-115.socal.res.rr.com) Quit (Quit: WeeChat 1.4)
[7:18] <Eduardo_> kefu, thank you. I need to buy more RAM though.
[7:18] <kefu> yw =)
[7:19] * avozza (~textual@a83-162-204-36.adsl.xs4all.nl) has joined #ceph
[7:20] * huangjun (~kvirc@113.57.168.154) has joined #ceph
[7:20] * ade (~abradshaw@dslb-088-075-182-015.088.075.pools.vodafone-ip.de) has joined #ceph
[7:22] * huangjun|2 (~kvirc@113.57.168.154) Quit (Ping timeout: 480 seconds)
[7:22] * bearkitten (~bearkitte@cpe-76-172-86-115.socal.res.rr.com) has joined #ceph
[7:24] * Jyron (~Moriarty@06SAAA5O6.tor-irc.dnsbl.oftc.net) Quit ()
[7:24] * RaidSoft (~drupal@tor-exit-1.netdive.xyz) has joined #ceph
[7:44] * derjohn_mobi (~aj@88.128.82.52) has joined #ceph
[7:47] * vicente (~~vicente@125-227-238-55.HINET-IP.hinet.net) has joined #ceph
[7:51] * b0e (~aledermue@p548FA508.dip0.t-ipconnect.de) has joined #ceph
[7:53] * Be-El (~blinke@nat-router.computational.bio.uni-giessen.de) has joined #ceph
[7:54] * RaidSoft (~drupal@6AGAAAP32.tor-irc.dnsbl.oftc.net) Quit ()
[7:54] * Gecko1986 (~Joppe4899@freedom.ip-eend.nl) has joined #ceph
[7:56] * haplo37 (~haplo37@107.190.32.70) Quit (Ping timeout: 480 seconds)
[7:58] * Eduardo__ (~Eduardo@85.193.28.37.rev.vodafone.pt) Quit (Ping timeout: 480 seconds)
[7:58] * Eduardo_ (~Eduardo@85.193.28.37.rev.vodafone.pt) Quit (Ping timeout: 480 seconds)
[8:02] * avozza (~textual@a83-162-204-36.adsl.xs4all.nl) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[8:10] * rakeshgm (~rakesh@121.244.87.117) has joined #ceph
[8:14] * rraja (~rraja@121.244.87.117) has joined #ceph
[8:14] * rraja_ (~rraja@121.244.87.117) has joined #ceph
[8:15] * EinstCra_ (~EinstCraz@58.247.117.134) has joined #ceph
[8:15] * EinstCrazy (~EinstCraz@58.247.117.134) Quit (Read error: Connection reset by peer)
[8:21] * Skaag (~lunix@cpe-172-91-77-84.socal.res.rr.com) has joined #ceph
[8:22] * Hemanth (~hkumar_@103.228.221.174) has joined #ceph
[8:23] * shyu (~Shanzhi@119.254.120.66) Quit (Remote host closed the connection)
[8:24] * Gecko1986 (~Joppe4899@6AGAAAP42.tor-irc.dnsbl.oftc.net) Quit ()
[8:24] * Inverness (~Sun7zu@207.244.70.35) has joined #ceph
[8:30] * dyasny (~dyasny@46-117-8-108.bb.netvision.net.il) Quit (Ping timeout: 480 seconds)
[8:31] * bvi (~bastiaan@185.56.32.1) has joined #ceph
[8:32] * shohn (~shohn@dslb-178-005-158-247.178.005.pools.vodafone-ip.de) has joined #ceph
[8:35] * EinstCra_ (~EinstCraz@58.247.117.134) Quit (Remote host closed the connection)
[8:35] * EinstCrazy (~EinstCraz@58.247.117.134) has joined #ceph
[8:37] * derjohn_mobi (~aj@88.128.82.52) Quit (Ping timeout: 480 seconds)
[8:37] * dugravot6 (~dugravot6@dn-infra-04.lionnois.site.univ-lorraine.fr) has joined #ceph
[8:39] * olid19811117 (~olid1982@ip503c3d0c.speed.planet.nl) has joined #ceph
[8:40] * Geph (~Geoffrey@41.77.153.99) has joined #ceph
[8:40] * RameshN (~rnachimu@121.244.87.117) Quit (Ping timeout: 480 seconds)
[8:43] * T1w (~jens@node3.survey-it.dk) has joined #ceph
[8:43] * dyasny (~dyasny@46-117-8-108.bb.netvision.net.il) has joined #ceph
[8:44] * avozza (~textual@a83-162-204-36.adsl.xs4all.nl) has joined #ceph
[8:45] * owasserm (~owasserm@2001:984:d3f7:1:5ec5:d4ff:fee0:f6dc) has joined #ceph
[8:45] * owasserm (~owasserm@2001:984:d3f7:1:5ec5:d4ff:fee0:f6dc) Quit (Read error: Connection reset by peer)
[8:46] * owasserm (~owasserm@2001:984:d3f7:1:5ec5:d4ff:fee0:f6dc) has joined #ceph
[8:50] * fsimonce (~simon@host201-70-dynamic.26-79-r.retail.telecomitalia.it) has joined #ceph
[8:51] * RameshN (~rnachimu@121.244.87.117) has joined #ceph
[8:52] * Miouge (~Miouge@94.136.92.20) has joined #ceph
[8:52] * olid19811117 (~olid1982@ip503c3d0c.speed.planet.nl) Quit (Ping timeout: 480 seconds)
[8:54] * Inverness (~Sun7zu@4MJAAD48W.tor-irc.dnsbl.oftc.net) Quit ()
[8:54] * Nephyrin (~Sami345@tor-exit.gansta93.com) has joined #ceph
[9:00] * shyu (~shyu@123.123.62.221) has joined #ceph
[9:03] * dgurtner (~dgurtner@87.215.61.26) has joined #ceph
[9:07] * evelu (~erwan@37.162.247.161) has joined #ceph
[9:15] * dgurtner_ (~dgurtner@82.199.64.68) has joined #ceph
[9:17] * dgurtner (~dgurtner@87.215.61.26) Quit (Ping timeout: 480 seconds)
[9:18] * evelu (~erwan@37.162.247.161) Quit (Read error: Connection reset by peer)
[9:19] <IvanJobs> I have a question about "ceph osd crush add ", I found this "osd crush add <osdname (id|osd.id)> add or update crushmap position and
[9:19] <IvanJobs> <float[0.0-]> <args> [<args>...] weight for <name> with <weight> and
[9:19] <IvanJobs> location <args>
[9:19] <IvanJobs> " in man doc. What doese <args> location means here?
[9:20] <IvanJobs> ceph osd crush add osd.3 1.0 <args>; what should I give for the last "<args>" ?
[9:24] * Nephyrin (~Sami345@06SAAA5QM.tor-irc.dnsbl.oftc.net) Quit ()
[9:24] * LorenXo (~Schaap@192.87.28.28) has joined #ceph
[9:26] * kawa2014 (~kawa@194.170.156.187) has joined #ceph
[9:27] * evelu (~erwan@37.162.247.161) has joined #ceph
[9:28] * analbeard (~shw@support.memset.com) has joined #ceph
[9:30] * derjohn_mobi (~aj@fw.gkh-setu.de) has joined #ceph
[9:31] * rmerritt_ (~rmerritt@lin-nyc.accolari.net) has joined #ceph
[9:31] * rmerritt (~rmerritt@lin-nyc.accolari.net) Quit (Read error: Connection reset by peer)
[9:33] * olid19811117 (~olid1982@ip503c3d0c.speed.planet.nl) has joined #ceph
[9:33] * Tene (~tene@173.13.139.236) Quit (Read error: Connection reset by peer)
[9:35] * Tene (~tene@173.13.139.236) has joined #ceph
[9:41] * wjw-freebsd (~wjw@smtp.digiware.nl) has joined #ceph
[9:41] <IvanJobs> ok, I figured it out. for my testing env, just use "root=default".
[9:44] * RameshN (~rnachimu@121.244.87.117) Quit (Ping timeout: 480 seconds)
[9:44] * goretoxo (~psilva@62.82.24.134.static.user.ono.com) has joined #ceph
[9:45] * olid19811117 (~olid1982@ip503c3d0c.speed.planet.nl) Quit (Ping timeout: 480 seconds)
[9:50] * rendar (~I@host159-27-dynamic.44-79-r.retail.telecomitalia.it) has joined #ceph
[9:50] * pabluk__ is now known as pabluk_
[9:51] * ToMiles (~ToMiles@medgents.ugent.be) has joined #ceph
[9:52] * RameshN (~rnachimu@121.244.87.117) has joined #ceph
[9:54] * LorenXo (~Schaap@76GAAECDA.tor-irc.dnsbl.oftc.net) Quit ()
[9:54] * poller (~FierceFor@exit1.ipredator.se) has joined #ceph
[10:02] * analbeard (~shw@support.memset.com) Quit (Quit: Leaving.)
[10:05] * linjan (~linjan@176.195.200.221) Quit (Ping timeout: 480 seconds)
[10:07] * analbeard (~shw@support.memset.com) has joined #ceph
[10:12] * jordanP (~jordan@204.13-14-84.ripe.coltfrance.com) has joined #ceph
[10:17] * IvanJobs (~hardes@103.50.11.146) Quit (Read error: Connection reset by peer)
[10:17] * IvanJobs (~hardes@103.50.11.146) has joined #ceph
[10:19] * efirs (~firs@c-50-185-70-125.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[10:21] * thomnico (~thomnico@gra94-13-78-225-161-58.fbx.proxad.net) has joined #ceph
[10:22] * bla (~b.laessig@chimeria.ext.pengutronix.de) Quit (Ping timeout: 480 seconds)
[10:23] * bla (~b.laessig@chimeria.ext.pengutronix.de) has joined #ceph
[10:23] * yanzheng (~zhyan@125.70.21.212) Quit (Read error: Connection reset by peer)
[10:24] * poller (~FierceFor@06SAAA5RZ.tor-irc.dnsbl.oftc.net) Quit ()
[10:24] * Zyn1 (~PappI@192.42.115.101) has joined #ceph
[10:25] * yanzheng (~zhyan@125.70.21.212) has joined #ceph
[10:26] * branto (~branto@178-253-130-11.3pp.slovanet.sk) has joined #ceph
[10:27] * LDA (~lda@host217-114-156-249.pppoe.mark-itt.net) has joined #ceph
[10:32] * Skaag (~lunix@cpe-172-91-77-84.socal.res.rr.com) Quit (Quit: Leaving.)
[10:33] * bara (~bara@ip4-83-240-10-82.cust.nbox.cz) has joined #ceph
[10:38] * bara (~bara@ip4-83-240-10-82.cust.nbox.cz) Quit (Quit: Bye guys! (??????????????????? ?????????)
[10:41] * Lea (~LeaChim@host86-159-239-193.range86-159.btcentralplus.com) has joined #ceph
[10:47] * rotbeard (~redbeard@185.32.80.238) has joined #ceph
[10:54] * Zyn1 (~PappI@76GAAECGF.tor-irc.dnsbl.oftc.net) Quit ()
[10:54] * MKoR (~djidis__@tor-exit1-readme.dfri.se) has joined #ceph
[10:54] <titzer> hey cephers
[10:56] <titzer> quick questions: data from the client hits the journal first, does the replication start from data in the journal or after the journal has been written to its storage drive?
[10:57] <smerz> ceph acknowledges a write once it's on all replica's journal
[10:57] <titzer> Also, does the replication hit the journal of the other osds or go directly into storage?
[10:57] <titzer> ah ok
[10:57] <titzer> so it hits say all three journals before writing it to the real storage drives?
[11:01] * Mik (~oftc-webi@129.102.242.53) has joined #ceph
[11:01] * Mik20 (~oftc-webi@129.102.242.53) has joined #ceph
[11:01] * Mik is now known as Guest182
[11:02] <Mik20> Hi ! Is possible to install Jewel with ceph-deploy on debian jessie ?
[11:03] <IcePic> titzer: it might start pushing from the first journal down to real storage while waiting for the other journal acks, but to the writing client, the important part is that it gets its ack only when all copies are acked by the respective OSD journals.
[11:06] <titzer> ah, I see
[11:07] <titzer> hm, would you expect to see more or less random iops performance going from a max_sync_interval of 1 sec to 10 sec?
[11:08] * T1w (~jens@node3.survey-it.dk) Quit (Ping timeout: 480 seconds)
[11:09] * Guest182 (~oftc-webi@129.102.242.53) Quit (Ping timeout: 480 seconds)
[11:11] <avozza> I use ceph-deploy osd create ceph01:sda:/dev/sdi but on the osd node the activation is stuck like : https://paste.fedoraproject.org/353852/65887146/
[11:12] <avozza> logs only containhttps://paste.fedoraproject.org/353853/03659611/
[11:12] <avozza> https://paste.fedoraproject.org/353853/03659611/
[11:14] <titzer> hm, have the lines where it says it failed?
[11:15] <titzer> if there are any such lines
[11:24] * MKoR (~djidis__@6AGAAAQDC.tor-irc.dnsbl.oftc.net) Quit ()
[11:24] * Linkshot (~PcJamesy@193.90.12.90) has joined #ceph
[11:24] <avozza> @titzer
[11:24] <avozza> there are none
[11:25] <avozza> but, in messages, a bunch of
[11:25] <avozza> https://paste.fedoraproject.org/353857/46036671/
[11:26] * bara (~bara@nat-pool-brq-t.redhat.com) has joined #ceph
[11:26] <avozza> i did wiped all disks with sgdisk -Z and sgdisk -o
[11:30] <titzer> oh, have not seen those before
[11:30] <titzer> I was going to suggest looking at the permissions your ceph user has on the journal partition, but that is probably not the issue
[11:31] <titzer> ceph should have given a permission error if that was the case
[11:31] <titzer> sorry
[11:32] <avozza> yeah I run as root
[11:32] <avozza> journal partition gets created
[11:32] <avozza> and data as well
[11:33] <titzer> and the journal symlink to the journal partition?
[11:34] * rraja_ (~rraja@121.244.87.117) Quit (Quit: Leaving)
[11:38] * thomnico (~thomnico@gra94-13-78-225-161-58.fbx.proxad.net) Quit (Remote host closed the connection)
[11:40] * redf (~red@chello080108089163.30.11.vie.surfer.at) Quit (Ping timeout: 480 seconds)
[11:40] * evelu (~erwan@37.162.247.161) Quit (Ping timeout: 480 seconds)
[11:43] * T1w (~jens@node3.survey-it.dk) has joined #ceph
[11:43] <avozza> there's no /var/lib/ceph/osd/ceph-0 !
[11:44] <avozza> it stops before that
[11:45] * ngoswami (~ngoswami@121.244.87.116) has joined #ceph
[11:48] * thomnico (~thomnico@gra94-13-78-225-161-58.fbx.proxad.net) has joined #ceph
[11:49] * evelu (~erwan@37.165.41.67) has joined #ceph
[11:50] <avozza> found the culprit
[11:50] <avozza> i remove the bonded interface, tried directly with just one interface, works
[11:51] * bjornar__ (~bjornar@109.247.131.38) has joined #ceph
[11:52] <titzer> that's odd, but good that you found the problem heh
[11:54] * Linkshot (~PcJamesy@6AGAAAQEQ.tor-irc.dnsbl.oftc.net) Quit ()
[11:58] <avozza> it was the MTU
[11:59] <avozza> I set MTU 9000 while the network wasn't ready
[11:59] <avozza> so probably was stuck contacting the monitors
[11:59] * redf (~red@chello080108089163.30.11.vie.surfer.at) has joined #ceph
[12:08] * shyu (~shyu@123.123.62.221) Quit (Ping timeout: 480 seconds)
[12:09] <ToMiles> Any reason why a ceph cluster in a HEALTH_OK state would continuously deep scrub the same 5 PG's ? did fs check no errors there nor with the disks. I've set the 'noscrub' flag for now.
[12:12] * thomnico (~thomnico@gra94-13-78-225-161-58.fbx.proxad.net) Quit (Quit: Ex-Chat)
[12:16] * avozza (~textual@a83-162-204-36.adsl.xs4all.nl) Quit (Quit: Textual IRC Client: www.textualapp.com)
[12:17] * Hemanth (~hkumar_@103.228.221.174) Quit (Ping timeout: 480 seconds)
[12:22] * wjw-freebsd (~wjw@smtp.digiware.nl) Quit (Ping timeout: 480 seconds)
[12:24] * Vale1 (~ylmson@tor-exit1-readme.dfri.se) has joined #ceph
[12:29] <Be-El> given a centos 7.2 host, which is the recommended kernel for an OSD host? the distribution kernel (3.10.0-327) or some kernel from a third-party repository?
[12:31] * EinstCrazy (~EinstCraz@58.247.117.134) Quit (Remote host closed the connection)
[12:32] <IvanJobs> ToMiles, weird. Maybe you should check config items related to deep scrubing, like "osd deep scrub interval".
[12:33] * Hemanth (~hkumar_@103.228.221.174) has joined #ceph
[12:34] <ToMiles> osd deep scrub interval isnt explicitly set to anything, so should be the defautl value
[12:36] <ToMiles> they were also only repeatedly scrubbing not deep scrubbing and only those 5 from the same pool
[12:37] * huangjun (~kvirc@113.57.168.154) Quit (Ping timeout: 480 seconds)
[12:39] * b0e (~aledermue@p548FA508.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[12:51] * Hemanth (~hkumar_@103.228.221.174) Quit (Quit: Leaving)
[12:51] * Hemanth (~hkumar_@103.228.221.174) has joined #ceph
[12:54] * Vale1 (~ylmson@6AGAAAQGT.tor-irc.dnsbl.oftc.net) Quit ()
[12:55] <rotbeard> just curious, what will happen if someone runs a productive cluster with "to few PGs"?
[12:55] <T1w> Be-El: for enterprise-level stuff I'd stick with the distributed version
[12:56] <T1w> Be-El: unless you have the experience to handle kernel upgrades properly and apart from regular yum-able updates
[12:56] <Be-El> T1w: the last time I've used a rpm-based distribution was >10 years ago
[12:56] <Be-El> so I better stick to the defaults
[12:57] <T1w> Be-El: for RHEL-like dists (including centos) I'd stick with defaults unless I have a very very good argument against..
[12:58] <T1w> right now I would really really REALLY like to have rsyslog v8 on my rhel7.2 machines, but I can see that redhat are planning a rebase to v8 in what looks like 7.3, so for now I'm holding off..
[12:59] * Wizeon (~nupanick@hessel3.torservers.net) has joined #ceph
[12:59] <T1w> I could easily install v8 from epel or rsyslogs own provided repo, but I'm sure it would conflict in some way from that redhat delivers
[12:59] <portante> T1w: it does not, really
[12:59] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[12:59] <portante> we run it locally for our logging environment
[12:59] <T1w> .. but then again it does..
[13:00] <portante> but you might want to wait for them to sort out libfastjson
[13:00] <T1w> daemon arguments are handled in a way the epel version doesn't
[13:00] <portante> they are working to clean up that library
[13:00] <T1w> mmm
[13:00] <portante> T1w: what do you mean?
[13:00] <portante> regarding daemon arguments?
[13:00] <portante> systemd.service file?
[13:00] <etienneme> rotbeard: a bad disk usage repartition and lower performances
[13:01] <T1w> contents in the /etc/sysconfig/rsyslog file is ignored and not used as a env config file in the .service file
[13:01] <portante> yes, the fedora/centos/rhel version does
[13:01] <portante> I have a COPR build of that
[13:01] <T1w> mmm - I don't want to do that stuff myself
[13:02] <T1w> so for now I'll just wait
[13:02] <portante> https://copr.fedorainfracloud.org/coprs/portante/rsyslog-8.17/
[13:02] <portante> you can at least try it out to see if it satisfies the systemd behavior you are looking for
[13:02] <rotbeard> etienneme, thanks for pointing out. if I build a new cluster (lets say with 100 OSDs), it is common to start up all OSDs and add pools with correct PG numbers later or add the pools with correct PG numbers first and then spawn up all OSDs?
[13:02] <T1w> ok, thanks
[13:02] <portante> once they settle out libfastjson, that will land in fedora (its in review)
[13:03] <T1w> portante: I only looked at the version available from http://www.rsyslog.com/rhelcentos-rpms/
[13:03] <T1w> .. or http://rpms.adiscon.com/v8-stable/
[13:03] <etienneme> rotbeard: I don't know if it's common, but I sometimes do it
[13:04] <portante> T1w: yes, so for some reason Adiscon builds their own version of Fedora/RHEL/CentOS and don't believe in syncing with the release its self
[13:04] <portante> I don't know why
[13:04] <portante> releases
[13:04] <etienneme> If I now my cluster will be x TB in few weeks, I set the right number of OSD/PG/PGP
[13:05] <T1w> portante: what are the issues with libfastjson? I've read the comments in bug 1313490, but I'm not sure I understand what the issue is - apart from a new dependency
[13:05] <T1w> portante: ok, nice to know
[13:06] <T1w> portante: once the rebase to v8 is complete should I expect coonflicts if I upgraded to v8 from COPR?
[13:07] * DanFoster (~Daniel@office.34sp.com) has joined #ceph
[13:07] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[13:07] <portante> none that I know of
[13:08] <portante> libfastjson is a replacement for json-c, but they have some other modules, liblognorm, I believe, which still depend on json-c, and the two libraries share the same name space
[13:08] <portante> so they are fixing that, and while they are at it cleaning up the code a bit.
[13:09] <rotbeard> etienneme, thanks
[13:09] * nardial (~ls@dslb-088-076-092-061.088.076.pools.vodafone-ip.de) has joined #ceph
[13:11] * vicente (~~vicente@125-227-238-55.HINET-IP.hinet.net) Quit (Quit: Leaving)
[13:19] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[13:26] * wyang (~wyang@114.111.166.43) has joined #ceph
[13:28] * Hemanth (~hkumar_@103.228.221.174) Quit (Ping timeout: 480 seconds)
[13:29] * Wizeon (~nupanick@6AGAAAQIO.tor-irc.dnsbl.oftc.net) Quit ()
[13:29] * Gecko1986 (~toast@199.68.196.126) has joined #ceph
[13:38] * thomnico (~thomnico@gra94-13-78-225-161-58.fbx.proxad.net) has joined #ceph
[13:40] <T1w> portante: sorry - was afk..
[13:41] <T1w> portante: ok, I'll take a look when I have the time - I have no (know) need for parsing json with rsyslog, so it's probably not an issue for me
[13:43] <T1w> I need the experimental feature reopenOnTruncate in imfile to solve a few things
[13:43] <portante> sounds good
[13:44] <T1w> the current workaround is a rsyslog restart at midnight to ensure truncated files are reopened.. and I'm sure some timing-issue could popup its ugly head at some point
[13:44] <T1w> timing issue such as a restart a few ms before the files is truncated..
[13:44] <T1w> file even
[13:45] <T1w> so changes are never picked up
[13:46] * nardial (~ls@dslb-088-076-092-061.088.076.pools.vodafone-ip.de) Quit (Quit: Leaving)
[13:48] * zhaochao (~zhaochao@125.39.9.144) Quit (Quit: ChatZilla 0.9.92 [Firefox 45.0.1/20160318172635])
[13:50] * thomnico (~thomnico@gra94-13-78-225-161-58.fbx.proxad.net) Quit (Ping timeout: 480 seconds)
[13:54] * overclk (~quassel@121.244.87.117) Quit (Remote host closed the connection)
[13:59] * Gecko1986 (~toast@06SAAA5V3.tor-irc.dnsbl.oftc.net) Quit ()
[13:59] * huangjun (~kvirc@117.151.43.17) has joined #ceph
[13:59] * dicko (~LRWerewol@6AGAAAQLN.tor-irc.dnsbl.oftc.net) has joined #ceph
[13:59] * dugravot61 (~dugravot6@dn-infra-04.lionnois.site.univ-lorraine.fr) has joined #ceph
[13:59] * dugravot6 (~dugravot6@dn-infra-04.lionnois.site.univ-lorraine.fr) Quit (Read error: Connection reset by peer)
[14:01] * Jeeves_ (~mark@2a03:7900:1:1:54f5:f6c9:69b6:3058) has joined #ceph
[14:01] <Jeeves_> Yo!
[14:02] <Jeeves_> Q: I just moved my journal from one SSD to another. Updated the symlink to the journal partition. All seems to be well. However, there also is a file journal_uuid, which contains the file of the previous journal.
[14:02] <Jeeves_> Can I just update that file? Who is using it?
[14:03] * dneary (~dneary@pool-96-237-170-97.bstnma.fios.verizon.net) Quit (Ping timeout: 480 seconds)
[14:04] * icey (~Chris@pool-74-109-7-163.phlapa.fios.verizon.net) has joined #ceph
[14:05] * dugravot6 (~dugravot6@dn-infra-04.lionnois.site.univ-lorraine.fr) has joined #ceph
[14:05] * dugravot61 (~dugravot6@dn-infra-04.lionnois.site.univ-lorraine.fr) Quit (Read error: Connection reset by peer)
[14:06] * wyang (~wyang@114.111.166.43) Quit (Quit: This computer has gone to sleep)
[14:06] * icey (~Chris@0001bbad.user.oftc.net) Quit (Remote host closed the connection)
[14:08] * itamarl (~itamarl@194.90.7.244) has joined #ceph
[14:09] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[14:09] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[14:11] * icey (~Chris@pool-74-109-7-163.phlapa.fios.verizon.net) has joined #ceph
[14:11] * yanzheng (~zhyan@125.70.21.212) Quit (Read error: Connection reset by peer)
[14:11] * icey (~Chris@0001bbad.user.oftc.net) Quit (Remote host closed the connection)
[14:11] * wyang (~wyang@122.225.69.14) has joined #ceph
[14:12] * yanzheng (~zhyan@125.70.21.212) has joined #ceph
[14:12] * fdmanana (~fdmanana@2001:8a0:6e0c:6601:f1df:eece:45c6:3086) has joined #ceph
[14:14] * dynamicudpate (~overonthe@199.68.193.62) Quit (Quit: Leaving)
[14:14] * rburkholder (~overonthe@199.68.193.62) Quit (Quit: Leaving)
[14:15] * icey (~Chris@pool-74-109-7-163.phlapa.fios.verizon.net) has joined #ceph
[14:18] <masterpe> Hi, I want to create an pool ssd-test on an separate ssd disk, But after creating the pool I get the following error: HEALTH_WARN 640 pgs degraded; 640 pgs stuck unclean; 640 pgs undersized
[14:18] <masterpe> Please see https://gist.github.com/mpiscaer/21b38abeab9497edee3996a9f7649e9e for more info
[14:18] <masterpe> Who can help me to solve this?
[14:20] <huangjun> how many replicas do you use?
[14:20] <Jeeves_> Undersized means there are too few osd
[14:21] <huangjun> maybe you have two datacenter and replica size is 3
[14:21] * fdmanana (~fdmanana@2001:8a0:6e0c:6601:f1df:eece:45c6:3086) Quit (Ping timeout: 480 seconds)
[14:24] <masterpe> 7root@ceph-mon-01-[LOC]:~# ceph osd pool get ssd-test size
[14:24] <masterpe> size: 2
[14:25] * shyu (~shyu@123.123.62.221) has joined #ceph
[14:26] <masterpe> I also added "ceph osd dump" to the gist
[14:27] <masterpe> I think I dat I make an error in the crush map
[14:27] <huangjun> step chooseleaf firstn 1 type datacenter
[14:27] <huangjun> this mean choose 1 item of datacenter,
[14:28] <huangjun> you can set to "step chooseleaf firstn 0 type datacenter"
[14:29] * dicko (~LRWerewol@6AGAAAQLN.tor-irc.dnsbl.oftc.net) Quit ()
[14:29] <huangjun> and your "min_size 0" which i dont understand, maybe you should set to "min_size 1"
[14:29] * Bromine (~Jaska@politkovskaja.torservers.net) has joined #ceph
[14:29] * dugravot6 (~dugravot6@dn-infra-04.lionnois.site.univ-lorraine.fr) Quit (Quit: Leaving.)
[14:30] <masterpe> I wanted to creaste aan cache with the ssd: http://docs.ceph.com/docs/master/rados/operations/cache-tiering/
[14:34] * shyu (~shyu@123.123.62.221) Quit (Ping timeout: 480 seconds)
[14:34] * EinstCrazy (~EinstCraz@116.225.239.138) has joined #ceph
[14:34] <huangjun> yes, you should create base pool and cache pool, then overlay cache pool to base pool
[14:36] <mistur> hello
[14:37] <titzer> Just for testing, with a max sync interval of 200 sec, min sync of 190 sec, 100GB ssd journal per OSD and running a random write test for two minutes, everything should hit the journals with no drops in performance, right?
[14:37] <titzer> hi
[14:37] * fdmanana (~fdmanana@2001:8a0:6e0c:6601:9c0a:2a4f:6369:c6) has joined #ceph
[14:37] * itamarl (~itamarl@194.90.7.244) Quit (Quit: itamarl)
[14:37] <mistur> how I can get the size of objects for an erasure pool ?
[14:38] <titzer> I reckon I might have to tweak the journal queue settings though
[14:38] <mistur> does object size depend on pg_num and pgp_num ?
[14:39] <huangjun> mistur: for ec pool the object size depends on origin data size and stipe_width
[14:40] <mistur> huangjun: can I get easly for a specific object ?
[14:40] <huangjun> titzer: yes, all writes operations are writeahead in journal for Filestore
[14:40] * bene2 (~bene@nat-pool-bos-t.redhat.com) has joined #ceph
[14:42] <huangjun> mistur: sorry, what do you mean?
[14:43] <mistur> I means I have 2 pools, one replicated and on erasure code
[14:44] <mistur> I push 8TB of data on each
[14:44] <mistur> 32768 files of 32MB (1TB)
[14:44] <mistur> 16384 files of 64MB (1TB)
[14:44] <mistur> etc
[14:45] <mistur> 256 files of 4GB (1TB)
[14:45] <mistur> I'd like to compare the size of object stored on disk
[14:46] * ira (~ira@nat-pool-bos-t.redhat.com) has joined #ceph
[14:46] <mistur> for files with differents size on the 2 pools
[14:47] <Kdecherf> hello world
[14:47] <Kdecherf> it seems that activate_osds ansible role activates nothing on Ubuntu 14.04 with infernalis
[14:47] <huangjun> mistur: ok, you use cephfs or rbd or rgw?
[14:48] <mistur> rgw
[14:49] <huangjun> do you enable radosstriper?
[14:50] <mistur> that's a good question, I never heard of that
[14:50] <mistur> :)
[14:50] * dneary (~dneary@nat-pool-bos-u.redhat.com) has joined #ceph
[14:50] * dugravot6 (~dugravot6@dn-infra-04.lionnois.site.univ-lorraine.fr) has joined #ceph
[14:51] <huangjun> if you are not enabled radosstriper, then the object size is the same of your file size for replicated pool,
[14:51] <huangjun> and the object size is file_size/K for erasure code pool,
[14:52] <mistur> ok
[14:52] <mistur> how I can check the status of radosstriper ?
[14:52] <huangjun> sorry, i'm not farmilar with it,
[14:53] <mistur> ok thanks
[14:53] <huangjun> you can check it on www.ceph.com
[14:53] <mistur> I gonna try to find this information
[14:53] <huangjun> :)
[14:59] * Bromine (~Jaska@06SAAA5XQ.tor-irc.dnsbl.oftc.net) Quit ()
[14:59] * toast (~aldiyen@s64.justhost.in.ua) has joined #ceph
[15:00] <titzer> huangjun: cheers
[15:01] <huangjun> :)
[15:04] <titzer> huangjun: http://i.imgur.com/b2FtabB.png
[15:04] <titzer> that's what I'm seeing though
[15:04] <titzer> I would have expected it to all stay steady up top
[15:05] * shyu (~shyu@123.123.62.221) has joined #ceph
[15:05] * m0zes (~mozes@129.130.32.11) Quit (Ping timeout: 480 seconds)
[15:06] * laevar (~jschulz1@134.76.80.11) has joined #ceph
[15:06] * sage (~quassel@2607:f298:6050:709d:dcee:8b17:54b4:7195) Quit (Remote host closed the connection)
[15:08] * sage (~quassel@2607:f298:6050:709d:ed7c:919e:58d5:5335) has joined #ceph
[15:08] * ChanServ sets mode +o sage
[15:08] <titzer> not sure what kind of buffer/parameter I'm hitting the limit of there
[15:08] <huangjun> titzer: how many osds?
[15:08] <titzer> 8
[15:08] <titzer> 4 nodes
[15:08] <titzer> 1 ssd in each node handling two journals each
[15:09] * kefu_ (~kefu@183.193.127.235) has joined #ceph
[15:09] * mattbenjamin (~mbenjamin@aa2.linuxbox.com) has joined #ceph
[15:09] <titzer> these are not the best ssds or controllers in the world, which is why the iops numbers are a bit low (also trying to figure out why we can't get peak performance out of them)
[15:09] <huangjun> 4k random writes?
[15:10] <titzer> yes
[15:10] * m0zes (~mozes@ns1.beocat.ksu.edu) has joined #ceph
[15:10] <titzer> 16 fio jobs, 64 QD
[15:10] * vikhyat is now known as vikhyat|brb
[15:10] <titzer> one such ssd alone can handle aprox 11000 random 4k write iops
[15:10] <titzer> in the cluster I'm seeing a total of like 3-400
[15:11] <titzer> with more or less default settings
[15:11] <huangjun> fio + rbd? or fio + fs?
[15:11] <titzer> rbd
[15:11] <titzer> no filesystem on the block device
[15:12] <huangjun> librbd or krbd?
[15:12] * rwheeler (~rwheeler@pool-173-48-195-215.bstnma.fios.verizon.net) has joined #ceph
[15:12] <titzer> ooh, good question
[15:12] <titzer> it's what a normal infernalis ceph installation would come with
[15:13] <titzer> filestore is 10k sas drives
[15:13] <huangjun> hardware is good though
[15:14] <huangjun> do you do 'rbd map'?
[15:14] <titzer> yes
[15:14] * aboyle (~aboyle__@ardent.csail.mit.edu) has joined #ceph
[15:15] <titzer> testing against a 100GB block device
[15:15] <huangjun> that is krbd (kernel rbd)
[15:15] <titzer> ah ok
[15:15] <titzer> better or worse than librbd?
[15:16] * kefu (~kefu@114.92.120.83) Quit (Ping timeout: 480 seconds)
[15:17] <huangjun> i dont know
[15:18] <laevar> hi, after upgrade and a toofull pg we are now in the situation that we have 1 inconsistent pg (30 scrub errors: ... is an unexpected clone) and we cannot map an rbd with the error "sysfs write failed, map failed: (110) Connection timed out". Can anyone give us a hint?
[15:18] * kefu (~kefu@183.193.127.235) has joined #ceph
[15:18] <huangjun> maybe you can change the fio queue depth
[15:18] <titzer> I suppose I should test that at some point
[15:18] <huangjun> yes, librbd have rbd cache which can improve iops
[15:18] <titzer> it's mostly the same results with 1, 16, 32, 64 QD
[15:19] <huangjun> what about "ceph osd perf" show?
[15:19] * shyu (~shyu@123.123.62.221) Quit (Remote host closed the connection)
[15:20] <titzer> http://pastie.org/10793487
[15:20] <huangjun> laevar: the inconsistent pg is belong to rbd pool?
[15:21] <laevar> huangjun: yes
[15:21] <huangjun> titzer: fs_commit_latency(ms) is perfect, which means your ssd is latency is very low,
[15:21] * kefu_ (~kefu@183.193.127.235) Quit (Ping timeout: 480 seconds)
[15:21] <huangjun> laevar: can you do 'ceph pg repair $PGID'
[15:22] <titzer> yeah
[15:22] <laevar> huangjun: i tried that already, sadly that changes nothing
[15:22] * kefu_ (~kefu@114.92.120.83) has joined #ceph
[15:23] <titzer> it's a bit extreme perhaps, but could you take down the osd containing the failing PG to force a rebuild of the OSD, then bring it back up after the cluster has sorted everything out?
[15:24] <huangjun> laevar: how about 'ceph pg deep-scrub $PGID'?
[15:25] <huangjun> if the inconsistent pg have the rbd_directory object, 'rbd map' will fail
[15:27] <laevar> huangjun: i had done a normal scrub, but did not started a deep-scrub. i started it now
[15:29] * toast (~aldiyen@6AGAAAQPM.tor-irc.dnsbl.oftc.net) Quit ()
[15:30] * kefu (~kefu@183.193.127.235) Quit (Ping timeout: 480 seconds)
[15:37] <laevar> huangjun: the result is the same
[15:38] <laevar> huangjun: 30 errors of "is unexpected clone"
[15:39] <laevar> huangjun: but at least good to know, that this is related to the rbd map..
[15:42] * wyang (~wyang@122.225.69.14) Quit (Quit: This computer has gone to sleep)
[15:44] * borun (~oftc-webi@178.237.98.13) has joined #ceph
[15:44] * wyang (~wyang@122.225.69.14) has joined #ceph
[15:44] <borun> hi there
[15:45] <borun> I got serious troubles getting radosgw to start
[15:45] * evelu (~erwan@37.165.41.67) Quit (Read error: Connection reset by peer)
[15:45] <borun> ceph version 0.80.11 (8424145d49264624a3b0a204aedb127835161070), process radosgw, pid 5761
[15:45] <borun> librados: client.radosgw.gateway authentication error (1) Operation not permitted
[15:45] <borun> Couldn't init storage provider (RADOS)
[15:45] <borun> I see that something about authorizations is missing, but I really can't see what
[15:47] <borun> plus logs don't say much more
[15:47] * shaunm (~shaunm@74.83.215.100) Quit (Ping timeout: 480 seconds)
[15:48] * kefu_ (~kefu@114.92.120.83) Quit (Max SendQ exceeded)
[15:48] * kefu (~kefu@li1072-18.members.linode.com) has joined #ceph
[15:49] * evelu (~erwan@37.165.41.67) has joined #ceph
[15:49] * Hemanth (~hkumar_@103.228.221.146) has joined #ceph
[15:51] <huangjun> laevar: can you paste out 'ceph -s'
[15:52] <borun> cluster a0307fbb-dca3-4551-9e9f-2b6aaa5b68c6 health HEALTH_WARN clock skew detected on mon.TIU-CEPH-002, mon.TIU-CEPH-003 Monitor clock skew detected monmap e1: 3 mons at {TIU-CEPH-001=10.66.60.131:6789/0,TIU-CEPH-002=10.66.60.132:6789/0,TIU-CEPH-003=10.66.60.133:6789/0}, election epoch 14, quorum 0,1,2 TIU-CEPH-001,TIU-CEPH-002,TIU-CEPH-003 osdmap e88: 9 osds: 9 up, 9 in pgmap v14634: 248 pgs, 10 pools, 1523 bytes data, 48 objects
[15:56] * lpabon (~quassel@nat-pool-bos-t.redhat.com) has joined #ceph
[15:58] * wjw-freebsd (~wjw@176.74.240.1) has joined #ceph
[15:58] <laevar> huangjun: http://pastebin.com/KcWRsZiq
[15:59] * Kidlvr (~BlS@tor.vpn-gateway.info) has joined #ceph
[16:01] <huangjun> laevar: ok, your cluster still in backfilling
[16:01] * xarses (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[16:02] * georgem (~Adium@206.108.127.16) has joined #ceph
[16:03] * karnan (~karnan@121.244.87.117) Quit (Quit: Leaving)
[16:05] * T1w (~jens@node3.survey-it.dk) Quit (Ping timeout: 480 seconds)
[16:06] <laevar> huangjun: yes, but the scrub-error will not vanish, right?
[16:07] * Hemanth (~hkumar_@103.228.221.146) Quit (Ping timeout: 480 seconds)
[16:07] <huangjun> yes, maybe you can wait the cluster backfill finish
[16:07] <huangjun> and then scrub\repair\deep-scrub again
[16:08] * yanzheng (~zhyan@125.70.21.212) Quit (Quit: This computer has gone to sleep)
[16:08] * haomaiwang (~haomaiwan@104.207.152.114) has joined #ceph
[16:08] <huangjun> borun: do you configured cephx for auth?
[16:08] <laevar> huangjun: ok, thanks, i will try this
[16:09] * haomaiwang (~haomaiwan@104.207.152.114) Quit (Remote host closed the connection)
[16:09] <huangjun> sorry with not much help, if you doubt the krbd, then you can try librbd
[16:09] * RameshN (~rnachimu@121.244.87.117) Quit (Ping timeout: 480 seconds)
[16:09] * haomaiwang (~haomaiwan@104.207.152.114) has joined #ceph
[16:09] * yanzheng (~zhyan@125.70.21.212) has joined #ceph
[16:09] * ircolle (~Adium@c-71-229-136-109.hsd1.co.comcast.net) has joined #ceph
[16:10] * haomaiwang (~haomaiwan@104.207.152.114) Quit (Remote host closed the connection)
[16:10] * haomaiwang (~haomaiwan@104.207.152.114) has joined #ceph
[16:11] * haomaiwang (~haomaiwan@104.207.152.114) Quit (Remote host closed the connection)
[16:11] * haomaiwang (~haomaiwan@104.207.152.114) has joined #ceph
[16:11] * etienneme (~arch@75.ip-167-114-228.eu) Quit (Ping timeout: 480 seconds)
[16:11] * ira (~ira@nat-pool-bos-t.redhat.com) Quit (Remote host closed the connection)
[16:12] * haomaiwang (~haomaiwan@104.207.152.114) Quit (Remote host closed the connection)
[16:12] * haomaiwang (~haomaiwan@104.207.152.114) has joined #ceph
[16:13] * haomaiwang (~haomaiwan@104.207.152.114) Quit (Remote host closed the connection)
[16:13] * haomaiwang (~haomaiwan@104.207.152.114) has joined #ceph
[16:13] * analbeard (~shw@support.memset.com) Quit (Quit: Leaving.)
[16:14] * haomaiwang (~haomaiwan@104.207.152.114) Quit (Remote host closed the connection)
[16:15] * brad_mssw (~brad@66.129.88.50) has joined #ceph
[16:17] * _28_ria (~kvirc@opfr028.ru) Quit (Ping timeout: 480 seconds)
[16:21] * vata1 (~vata@207.96.182.162) has joined #ceph
[16:21] <Mik20> When i mount a fs, the vm freeze and i don't know why : mount -t ceph 192.168.1.2:6789:/ /mnt/cephfs -o name=admin,secretfile=admin.secret
[16:21] <Mik20> i have to reboot the vm
[16:23] * vikhyat|brb (~vumrao@121.244.87.116) Quit (Quit: Leaving)
[16:25] <boolman> is your mds cluster healthy?
[16:26] * linjan (~linjan@176.195.200.221) has joined #ceph
[16:27] * gregmark (~Adium@68.87.42.115) has joined #ceph
[16:28] * linjan (~linjan@176.195.200.221) Quit ()
[16:28] * linjan (~linjan@freedom.ip-eend.nl) has joined #ceph
[16:28] * overclk (~quassel@117.202.102.64) has joined #ceph
[16:29] * Kidlvr (~BlS@4MJAAD5KL.tor-irc.dnsbl.oftc.net) Quit ()
[16:29] * pakman__ (~Quackie@76GAAECZF.tor-irc.dnsbl.oftc.net) has joined #ceph
[16:34] * haplo37 (~haplo37@199.91.185.156) has joined #ceph
[16:34] * etienneme (~arch@75.ip-167-114-228.eu) has joined #ceph
[16:37] * shaunm (~shaunm@cpe-192-180-18-62.kya.res.rr.com) has joined #ceph
[16:38] * wernerru (~oftc-webi@dev.elc.msu.edu) has joined #ceph
[16:40] * ira (~ira@nat-pool-bos-t.redhat.com) has joined #ceph
[16:41] * Hemanth (~hkumar_@103.228.221.146) has joined #ceph
[16:41] <huangjun> where to download April CDM video ?
[16:43] * wushudoin (~wushudoin@2601:646:8201:7769:2ab2:bdff:fe0b:a6ee) has joined #ceph
[16:44] * sudocat (~dibarra@2602:306:8bc7:4c50:41b3:a4af:5988:bef) Quit (Ping timeout: 480 seconds)
[16:44] <Be-El> how do I change the IP address of an OSD host? shutting down the OSD, change the address and restart the OSDs?
[16:44] * wushudoin (~wushudoin@2601:646:8201:7769:2ab2:bdff:fe0b:a6ee) Quit ()
[16:44] * wushudoin (~wushudoin@2601:646:8201:7769:2ab2:bdff:fe0b:a6ee) has joined #ceph
[16:45] * yanzheng (~zhyan@125.70.21.212) Quit (Quit: This computer has gone to sleep)
[16:45] <m0zes> as long as it is still in the networks defined in your ceph.conf, that should be all you need to do.
[16:46] * rakeshgm (~rakesh@121.244.87.117) Quit (Quit: Leaving)
[16:47] * bara (~bara@nat-pool-brq-t.redhat.com) Quit (Quit: Bye guys! (??????????????????? ?????????)
[16:49] <TMM> m0zes, it appears my issues were caused by numad btw. It was wildly migrating large vm instances between numa nodes while the osds were spawning threads during peering
[16:49] <TMM> m0zes, it seems that the slow requests were actually caused by the instances doing the requests rather than the osds being too slow
[16:49] * kefu (~kefu@li1072-18.members.linode.com) Quit (Read error: Connection reset by peer)
[16:50] <TMM> m0zes, I suppose nobody can really tell the difference between a slow client and a slow osd, the work remains on the osd's queue
[16:53] * togdon (~togdon@74.121.28.6) has joined #ceph
[16:54] <wernerru> i remember seeing a changelog a ways back about setting mds_max_file_size - is that something we can adjust now on the fly, or does that still require killing the mds (and 35tb of data haha) and starting a new cephfs...
[16:54] * kefu (~kefu@114.92.120.83) has joined #ceph
[16:54] * vbellur (~vijay@71.234.224.255) Quit (Ping timeout: 480 seconds)
[16:58] * wjw-freebsd2 (~wjw@176.74.240.1) has joined #ceph
[16:59] * pakman__ (~Quackie@76GAAECZF.tor-irc.dnsbl.oftc.net) Quit ()
[16:59] * Doodlepieguy (~ricin@6.tor.exit.babylon.network) has joined #ceph
[17:01] * wjw-freebsd (~wjw@176.74.240.1) Quit (Ping timeout: 480 seconds)
[17:06] * vicente (~vicente@111-241-31-95.dynamic.hinet.net) has joined #ceph
[17:09] * reed (~reed@75-101-54-18.dsl.static.fusionbroadband.com) has joined #ceph
[17:10] * jdillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) has joined #ceph
[17:11] * bjornar__ (~bjornar@109.247.131.38) Quit (Ping timeout: 480 seconds)
[17:13] * xarses (~xarses@64.124.158.100) has joined #ceph
[17:15] * ade_b (~abradshaw@dslb-188-106-099-129.188.106.pools.vodafone-ip.de) has joined #ceph
[17:15] * dgurtner (~dgurtner@87.215.61.26) has joined #ceph
[17:17] * dgurtner_ (~dgurtner@82.199.64.68) Quit (Ping timeout: 480 seconds)
[17:21] * The_Ball (~pi@20.92-221-43.customer.lyse.net) Quit (Read error: Connection reset by peer)
[17:21] * The_Ball (~pi@20.92-221-43.customer.lyse.net) has joined #ceph
[17:21] * ade (~abradshaw@dslb-088-075-182-015.088.075.pools.vodafone-ip.de) Quit (Ping timeout: 480 seconds)
[17:22] * ircolle1 (~Adium@c-71-229-136-109.hsd1.co.comcast.net) has joined #ceph
[17:23] * jclm (~jclm@ip68-96-198-45.lv.lv.cox.net) has joined #ceph
[17:23] * DV (~veillard@2001:41d0:a:f29f::1) Quit (Remote host closed the connection)
[17:23] * wyang (~wyang@122.225.69.14) Quit (Quit: This computer has gone to sleep)
[17:25] * dgurtner_ (~dgurtner@82.199.64.68) has joined #ceph
[17:25] * sudocat (~dibarra@192.185.1.20) has joined #ceph
[17:26] * huangjun (~kvirc@117.151.43.17) Quit (Ping timeout: 480 seconds)
[17:26] * DV (~veillard@2001:41d0:a:f29f::1) has joined #ceph
[17:26] * khyron (~khyron@187.190.152.61) Quit (Quit: The computer fell asleep)
[17:26] * dgurtner (~dgurtner@87.215.61.26) Quit (Ping timeout: 480 seconds)
[17:27] * vbellur (~vijay@nat-pool-bos-u.redhat.com) has joined #ceph
[17:27] * olid19811117 (~olid1982@ip503c3d0c.speed.planet.nl) has joined #ceph
[17:29] * ircolle (~Adium@c-71-229-136-109.hsd1.co.comcast.net) Quit (Ping timeout: 480 seconds)
[17:29] * Doodlepieguy (~ricin@76GAAEC1X.tor-irc.dnsbl.oftc.net) Quit ()
[17:29] * zapu (~Coestar@176.10.99.205) has joined #ceph
[17:33] * wernerru (~oftc-webi@dev.elc.msu.edu) Quit (Quit: Page closed)
[17:33] * wernerru (~oftc-webi@dev.elc.msu.edu) has joined #ceph
[17:39] * wjw-freebsd2 (~wjw@176.74.240.1) Quit (Ping timeout: 480 seconds)
[17:43] * joshd1 (~jdurgin@68-119-140-18.dhcp.ahvl.nc.charter.com) has joined #ceph
[17:43] * blynch (~blynch@vm-nat.msi.umn.edu) has joined #ceph
[17:48] * Mik20 (~oftc-webi@129.102.242.53) Quit (Quit: Page closed)
[17:50] * wernerru (~oftc-webi@dev.elc.msu.edu) Quit (Quit: Page closed)
[17:50] * wernerru (~oftc-webi@dev.elc.msu.edu) has joined #ceph
[17:51] * rraja (~rraja@121.244.87.117) Quit (Quit: Leaving)
[17:51] * swami1 (~swami@49.32.0.102) Quit (Quit: Leaving.)
[17:53] * alexxy (~alexxy@biod.pnpi.spb.ru) Quit (Remote host closed the connection)
[17:54] * Skaag (~lunix@65.200.54.234) has joined #ceph
[17:54] * alexxy (~alexxy@biod.pnpi.spb.ru) has joined #ceph
[17:58] * Geph (~Geoffrey@41.77.153.99) Quit (Ping timeout: 480 seconds)
[17:59] * zapu (~Coestar@6AGAAAQ0Y.tor-irc.dnsbl.oftc.net) Quit ()
[17:59] * togdon (~togdon@74.121.28.6) Quit (Quit: Bye-Bye.)
[18:00] * wernerru (~oftc-webi@dev.elc.msu.edu) Quit (Quit: Page closed)
[18:05] * rotbeard (~redbeard@185.32.80.238) Quit (Quit: Leaving)
[18:06] * ToMiles (~ToMiles@medgents.ugent.be) Quit (Ping timeout: 480 seconds)
[18:07] * b0e (~aledermue@p54AFFDE7.dip0.t-ipconnect.de) has joined #ceph
[18:08] * dugravot6 (~dugravot6@dn-infra-04.lionnois.site.univ-lorraine.fr) Quit (Quit: Leaving.)
[18:11] * jordanP (~jordan@204.13-14-84.ripe.coltfrance.com) Quit (Quit: Leaving)
[18:11] * thomnico (~thomnico@37.163.232.65) has joined #ceph
[18:12] * davidzlap (~Adium@2605:e000:1313:8003:f4ed:59c6:2a5f:f77b) has joined #ceph
[18:16] <Kdecherf> does anyone use block devices as journals on infernalis?
[18:16] * Kioob1 (~Kioob@ALyon-652-1-45-46.w109-213.abo.wanadoo.fr) has joined #ceph
[18:16] <Kdecherf> i have a permission denied when using ceph-disk activate on the journal file
[18:18] * pabluk_ is now known as pabluk__
[18:18] <Be-El> Kdecherf: does the ceph user has the correct permissions for accessing the journal device (infernalis or jewel release)?
[18:19] * Kioob1 (~Kioob@ALyon-652-1-45-46.w109-213.abo.wanadoo.fr) Quit (Remote host closed the connection)
[18:21] <Kdecherf> Be-El: i don't think so, i wonder if ceph-disk prepare did its job correctly
[18:22] * vicente (~vicente@111-241-31-95.dynamic.hinet.net) Quit (Ping timeout: 480 seconds)
[18:22] <Be-El> Kdecherf: i don't know whether it is done by ceph-disk or by the udev rules
[18:22] * IvanJobs (~hardes@103.50.11.146) Quit (Ping timeout: 480 seconds)
[18:23] <m0zes> the permissions are done by udev, if ceph-disk did its job properly.
[18:23] <Kdecherf> Be-El: i don't have any udev rule, the disk was originally prepared by ceph-disk prepare (ansible playbook)
[18:23] * thomnico (~thomnico@37.163.232.65) Quit (Quit: Ex-Chat)
[18:23] <Kdecherf> well, so i missed something somewhere
[18:23] <Kdecherf> the journal device was not empty during ceph-disk prepare
[18:23] <m0zes> the journal device should be gpt partitioned, with the parttype-uuid set to '45b0969e-9b03-4f30-b4c6-b4b80ceff106'
[18:24] <Kdecherf> hm ok, i will destroy the partition tables before retrying
[18:24] <m0zes> you should be able to change the parttype-uuid with sgdisk.
[18:24] <m0zes> or at least check them
[18:24] * ngoswami (~ngoswami@121.244.87.116) Quit (Quit: Leaving)
[18:25] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit (Ping timeout: 480 seconds)
[18:28] * zaitcev (~zaitcev@c-50-130-189-82.hsd1.nm.comcast.net) has joined #ceph
[18:29] * Tarazed (~JamesHarr@65.19.167.132) has joined #ceph
[18:29] * kawa2014 (~kawa@194.170.156.187) Quit (Quit: Leaving)
[18:30] * Be-El (~blinke@nat-router.computational.bio.uni-giessen.de) Quit (Quit: Leaving.)
[18:30] * thomnico_ (~thomnico@37.163.232.65) has joined #ceph
[18:34] * goretoxo (~psilva@62.82.24.134.static.user.ono.com) Quit (Ping timeout: 480 seconds)
[18:36] * shylesh__ (~shylesh@59.95.71.149) has joined #ceph
[18:38] * vasu (~vasu@c-73-231-60-138.hsd1.ca.comcast.net) has joined #ceph
[18:43] * thomnico (~thomnico@37.163.232.65) has joined #ceph
[18:44] * thomnico_ (~thomnico@37.163.232.65) Quit (Ping timeout: 480 seconds)
[18:46] * dscastro (~dscastro@145-96-231-201.fibertel.com.ar) has joined #ceph
[18:46] * dscastro (~dscastro@145-96-231-201.fibertel.com.ar) Quit (autokilled: This host may be infected. Mail support@oftc.net with questions. BOPM (2016-04-11 16:46:39))
[18:48] * Axion_Joey (~oftc-webi@static-108-47-170-18.lsanca.fios.frontiernet.net) has joined #ceph
[18:48] <Axion_Joey> Hi Guys
[18:48] <Axion_Joey> I've got an architecture question
[18:49] <Axion_Joey> We have ceph deployed at 3 Data Centers
[18:49] * joshd1 (~jdurgin@68-119-140-18.dhcp.ahvl.nc.charter.com) Quit (Quit: Leaving.)
[18:49] * mrclong (~clong@4.30.56.154) has joined #ceph
[18:49] <Axion_Joey> right now each DC is stand alone. We have 3 ceph mon's and 2 ceph osd servers per data center
[18:49] <Axion_Joey> We're rolling out a new application that needs near real time storage synchronization per dc
[18:50] <Axion_Joey> But the challenge is that the app is latency sensitive
[18:50] * vikhyat (~vumrao@114.143.41.211) has joined #ceph
[18:50] <Axion_Joey> Is it possible to configure ceph to read and write to local servers and then have those server synchronize with the other data centers?
[18:51] * thomnico (~thomnico@37.163.232.65) Quit (Ping timeout: 480 seconds)
[18:51] <m0zes> seperate pools? or is all the data needed at both data centers?
[18:51] <Axion_Joey> for this app all of the data is needed at all three data centers
[18:52] <m0zes> then no.
[18:52] <Axion_Joey> that was easy. thanks :)
[18:52] * bvi (~bastiaan@185.56.32.1) Quit (Ping timeout: 480 seconds)
[18:52] <m0zes> reads come off primary osds. writes go to primary osds to be replicated out.
[18:53] <Axion_Joey> can I configure the primary osd's differently at each physical location for 1 pool?
[18:54] <m0zes> no. the primaries are on a per pg basis.
[18:54] <Axion_Joey> got it.
[18:55] <Axion_Joey> can ceph do async replication between pools?
[18:56] <Axion_Joey> I saw an article about that but it was from back in 2013
[18:58] * rakeshgm (~rakesh@106.51.225.4) has joined #ceph
[18:59] * Tarazed (~JamesHarr@76GAAEC7U.tor-irc.dnsbl.oftc.net) Quit ()
[18:59] * DV (~veillard@2001:41d0:a:f29f::1) Quit (Remote host closed the connection)
[19:00] * DV (~veillard@2001:41d0:a:f29f::1) has joined #ceph
[19:00] * DV (~veillard@2001:41d0:a:f29f::1) Quit (Remote host closed the connection)
[19:01] * DV (~veillard@2001:41d0:a:f29f::1) has joined #ceph
[19:02] * olid19811117 (~olid1982@ip503c3d0c.speed.planet.nl) Quit (Ping timeout: 480 seconds)
[19:07] * kefu (~kefu@114.92.120.83) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[19:07] * Eduardo__ (~Eduardo@85.193.28.37.rev.vodafone.pt) has joined #ceph
[19:08] * Eduardo_ (~Eduardo@85.193.28.37.rev.vodafone.pt) has joined #ceph
[19:08] * sjackson (~sjackson@207.111.246.196) Quit (Remote host closed the connection)
[19:09] * laevar (~jschulz1@134.76.80.11) Quit (Ping timeout: 480 seconds)
[19:09] * ade_b (~abradshaw@dslb-188-106-099-129.188.106.pools.vodafone-ip.de) Quit (Ping timeout: 480 seconds)
[19:11] * dgurtner_ (~dgurtner@82.199.64.68) Quit (Ping timeout: 480 seconds)
[19:12] * laevar (~jschulz1@134.76.80.11) has joined #ceph
[19:15] * vicente (~vicente@111-241-31-95.dynamic.hinet.net) has joined #ceph
[19:15] * Eduardo__ (~Eduardo@85.193.28.37.rev.vodafone.pt) Quit (Ping timeout: 480 seconds)
[19:16] * Eduardo_ (~Eduardo@85.193.28.37.rev.vodafone.pt) Quit (Ping timeout: 480 seconds)
[19:18] * rotbeard (~redbeard@2a02:908:df15:1800:6267:20ff:feb7:c20) has joined #ceph
[19:20] * Vacuum_ (~Vacuum@88.130.212.12) has joined #ceph
[19:22] * rakeshgm (~rakesh@106.51.225.4) Quit (Read error: No route to host)
[19:23] * b0e (~aledermue@p54AFFDE7.dip0.t-ipconnect.de) Quit (Quit: Leaving.)
[19:24] * vicente (~vicente@111-241-31-95.dynamic.hinet.net) Quit (Ping timeout: 480 seconds)
[19:24] * funnel (~funnel@0001c7d4.user.oftc.net) Quit (Remote host closed the connection)
[19:25] * lcurtis (~lcurtis@47.19.105.250) has joined #ceph
[19:26] * funnel (~funnel@0001c7d4.user.oftc.net) has joined #ceph
[19:27] * Vacuum__ (~Vacuum@88.130.220.135) Quit (Ping timeout: 480 seconds)
[19:29] * AluAlu (~FierceFor@ns330209.ip-5-196-66.eu) has joined #ceph
[19:29] * mykola (~Mikolaj@91.245.78.165) has joined #ceph
[19:29] * derjohn_mobi (~aj@fw.gkh-setu.de) Quit (Ping timeout: 480 seconds)
[19:31] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[19:31] * funnel (~funnel@0001c7d4.user.oftc.net) Quit (Remote host closed the connection)
[19:31] * funnel (~funnel@0001c7d4.user.oftc.net) has joined #ceph
[19:33] * DanFoster (~Daniel@office.34sp.com) Quit (Quit: Leaving)
[19:33] * overclk (~quassel@117.202.102.64) Quit (Remote host closed the connection)
[19:34] * ade_b (~abradshaw@dslb-188-106-099-129.188.106.pools.vodafone-ip.de) has joined #ceph
[19:34] * ade_b (~abradshaw@dslb-188-106-099-129.188.106.pools.vodafone-ip.de) Quit ()
[19:39] * thomnico (~thomnico@62.119.166.9) has joined #ceph
[19:43] * ira (~ira@nat-pool-bos-t.redhat.com) Quit (Quit: Leaving)
[19:44] * thomnico_ (~thomnico@37.163.232.65) has joined #ceph
[19:45] * infernix (~nix@spirit.infernix.net) Quit (Ping timeout: 480 seconds)
[19:46] * infernix (nix@2001:41f0::2) has joined #ceph
[19:49] * thomnico (~thomnico@62.119.166.9) Quit (Ping timeout: 480 seconds)
[19:56] * ToMiles (~ToMiles@91.179.71.93) has joined #ceph
[19:59] * AluAlu (~FierceFor@4MJAAD5RB.tor-irc.dnsbl.oftc.net) Quit ()
[19:59] * theghost99 (~straterra@tor.yrk.urgs.uk0.bigv.io) has joined #ceph
[20:06] * bvi (~bastiaan@152-64-132-5.ftth.glasoperator.nl) has joined #ceph
[20:09] * vikhyat (~vumrao@114.143.41.211) Quit (Quit: Leaving)
[20:11] * bvi (~bastiaan@152-64-132-5.ftth.glasoperator.nl) Quit (Remote host closed the connection)
[20:13] * bjornar__ (~bjornar@ti0099a430-1561.bb.online.no) has joined #ceph
[20:14] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[20:16] * ToMiles (~ToMiles@91.179.71.93) Quit (Quit: leaving)
[20:16] * vicente (~vicente@111-241-31-95.dynamic.hinet.net) has joined #ceph
[20:24] * olid19811117 (~olid1982@ip503c3d0c.speed.planet.nl) has joined #ceph
[20:24] * thomnico_ (~thomnico@37.163.232.65) Quit (Ping timeout: 480 seconds)
[20:24] * vicente (~vicente@111-241-31-95.dynamic.hinet.net) Quit (Ping timeout: 480 seconds)
[20:28] * TomasCZ (~TomasCZ@yes.tenlab.net) has joined #ceph
[20:29] * theghost99 (~straterra@7V7AAD3WI.tor-irc.dnsbl.oftc.net) Quit ()
[20:29] * AG_Scott (~Keiya@edwardsnowden1.torservers.net) has joined #ceph
[20:29] * shylesh__ (~shylesh@59.95.71.149) Quit (Remote host closed the connection)
[20:30] <wolsen> has anyone seen an issue where an osd was removed from a cluster but the device id remains in as device <id> device<id> in the crushmap?
[20:32] * skinnejo (~skinnejo@162.247.128.73) has joined #ceph
[20:33] * thomnico__ (~thomnico@37.164.61.176) has joined #ceph
[20:34] <m0zes> they are placeholders. they'll be re-used when creating new osds, now that you've deleted the old ones.
[20:35] * swami1 (~swami@27.7.161.87) has joined #ceph
[20:35] * thomnico__ (~thomnico@37.164.61.176) Quit ()
[20:35] * thomnico (~thomnico@37.164.61.176) has joined #ceph
[20:36] * infernix (nix@2001:41f0::2) Quit (Ping timeout: 480 seconds)
[20:38] * DV (~veillard@2001:41d0:a:f29f::1) Quit (Remote host closed the connection)
[20:38] <wolsen> m0zes, hmm when we recreated a new osd it picked a new id from what I can tell
[20:38] * DV (~veillard@2001:41d0:a:f29f::1) has joined #ceph
[20:38] <m0zes> did you delete the old one before creating a new one?
[20:39] <wolsen> I didn't do the deletion myself, I'm coming into it
[20:39] * Discovery (~Discovery@178.239.49.71) has joined #ceph
[20:39] <wolsen> so I can't vouch for the specific order
[20:39] <m0zes> it chooses the next free id when adding an osd.
[20:39] <wolsen> ok, so those are now free for selection again?
[20:39] <m0zes> yep
[20:40] * neurodrone_ (~neurodron@static-108-29-37-177.nycmny.fios.verizon.net) has joined #ceph
[20:41] <laevar> i have 30 scrub errors with "is an unexpected clone" and tried nearly everything i know of besides making ceph pg force_create_pg . I am not quite sure what happens in this case (i do have replica-size of 3). Perhaps you have an idea what would be the best way to come out of this?
[20:42] <wolsen> thxx m0zes
[20:43] * neurodrone (~neurodron@158.106.193.162) Quit (Ping timeout: 480 seconds)
[20:43] * neurodrone_ is now known as neurodrone
[20:46] * med (~medberry@71.74.177.250) Quit (Quit: Coyote finally caught me)
[20:47] * thomnico (~thomnico@37.164.61.176) Quit (Ping timeout: 480 seconds)
[20:48] <wolsen> m0zes, I'm curious - I've got an incomplete PG, with the pg dump saying osd xy is a down osd we would probe - however osd xy was one of those that were deleted
[20:49] * infernix (nix@2001:41f0::2) has joined #ceph
[20:49] <T1> Axion_Joey: what you are asking for is comming in new releases
[20:51] <T1> or.. at least parts are
[20:51] <T1> http://www.sebastien-han.fr/blog/2016/03/28/ceph-jewel-preview-ceph-rbd-mirroring/
[20:54] * med (~medberry@71.74.177.250) has joined #ceph
[20:59] * AG_Scott (~Keiya@06SAAA569.tor-irc.dnsbl.oftc.net) Quit ()
[20:59] * xanax` (~ylmson@76GAAEDF2.tor-irc.dnsbl.oftc.net) has joined #ceph
[21:01] * olid19811117 (~olid1982@ip503c3d0c.speed.planet.nl) Quit (Ping timeout: 480 seconds)
[21:05] * DV (~veillard@2001:41d0:a:f29f::1) Quit (Remote host closed the connection)
[21:06] <Axion_Joey> I've got a performance problem. I have a system set up in our lab that had 2 OSD servers with 3 ssd's in them each
[21:07] <Axion_Joey> According to rados bench the max bandwidth was 200MB'sec
[21:07] * DV (~veillard@2001:41d0:a:f29f::1) has joined #ceph
[21:07] <Axion_Joey> I just added 3 more ssd's to each server, and re-ran rados bench and the bandwidth hasn't changed
[21:07] <Axion_Joey> shouldn't I have seen some improvement with adding 6 more disks?
[21:09] <georgem> Axion_Joey: related to your earlier question, you can make an OSD less likely to be chosen as primary http://ceph.com/planet/ceph-get-the-best-of-your-ssd-with-primary-affinity/
[21:09] <Axion_Joey> georgem that's awesome!
[21:09] * csoukup (~csoukup@159.140.254.99) has joined #ceph
[21:10] <georgem> Axion_Joey: but that only helps you with the reads
[21:10] * Hemanth_ (~hkumar_@103.228.221.142) has joined #ceph
[21:10] * Cassiopaya (~Discovery@ip-176-199-137-160.hsi06.unitymediagroup.de) has joined #ceph
[21:10] <Axion_Joey> Yeah I'm reading that now
[21:11] * olid19811117 (~olid1982@ip503c3d0c.speed.planet.nl) has joined #ceph
[21:11] * chasmo77 (~chas77@158.183-62-69.ftth.swbr.surewest.net) Quit (Quit: It's just that easy)
[21:11] * Discovery (~Discovery@178.239.49.71) Quit (Ping timeout: 480 seconds)
[21:13] * Hemanth (~hkumar_@103.228.221.146) Quit (Ping timeout: 480 seconds)
[21:14] <Axion_Joey> in general should ceph's write speed increase by adding additional drives to existing hosts? or do I need to add more osd hosts to see an improvement?
[21:15] <davidzlap> laevar: Deleting the unexpected clones would resolve the problem. It is most likely that those clones didn't get removed for some reason. If so, stop the OSD and use ceph-objectstore-tool to remove those 30 objects.
[21:17] * vicente (~vicente@111-241-31-95.dynamic.hinet.net) has joined #ceph
[21:23] * csoukup (~csoukup@159.140.254.99) Quit (Ping timeout: 480 seconds)
[21:24] <Axion_Joey> More info. I have a 10 gig network. the osd servers only show 2% wait time
[21:25] * vicente (~vicente@111-241-31-95.dynamic.hinet.net) Quit (Ping timeout: 480 seconds)
[21:25] <Axion_Joey> there are 6 ceph-osd daemons running on the osd server while the test is running and none of them show more than 25% cpu usage
[21:25] * lpabon (~quassel@nat-pool-bos-t.redhat.com) Quit (Remote host closed the connection)
[21:28] * ira (~ira@c-24-34-255-34.hsd1.ma.comcast.net) has joined #ceph
[21:29] * xanax` (~ylmson@76GAAEDF2.tor-irc.dnsbl.oftc.net) Quit ()
[21:29] * Uniju1 (~DoDzy@76GAAEDHN.tor-irc.dnsbl.oftc.net) has joined #ceph
[21:30] * thomnico (~thomnico@175.233-245-81.adsl-dyn.isp.belgacom.be) has joined #ceph
[21:33] * thomnico is now known as thomnico|Brussels
[21:41] * swami1 (~swami@27.7.161.87) Quit (Quit: Leaving.)
[21:43] * xdr (67334b61@107.161.19.53) has joined #ceph
[21:43] <xdr> Hi ! I have a non-retina Mac book Pro which is upgraded to 16 gb RAM . it has a 250 gb ssd + 500gb hdd . runs a core i5 @ 2.7 ghz and intel hd 4000 as its gpu. I would like to know if i can build ceph on this config if i have a linux distro installed on a partition .Linux would be using all these configs as it is dual booted . need help here . than
[21:43] <xdr> ks .
[21:45] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[21:47] * rendar (~I@host159-27-dynamic.44-79-r.retail.telecomitalia.it) Quit (Ping timeout: 480 seconds)
[21:49] * rendar (~I@host159-27-dynamic.44-79-r.retail.telecomitalia.it) has joined #ceph
[21:49] <xdr> ??
[21:50] * vbellur (~vijay@nat-pool-bos-u.redhat.com) Quit (Ping timeout: 480 seconds)
[21:55] * rotbeard (~redbeard@2a02:908:df15:1800:6267:20ff:feb7:c20) Quit (Quit: Leaving)
[21:56] * dkrdkr (48a3dc03@107.161.19.53) has joined #ceph
[21:59] * Uniju1 (~DoDzy@76GAAEDHN.tor-irc.dnsbl.oftc.net) Quit ()
[21:59] * Cassiopaya (~Discovery@ip-176-199-137-160.hsi06.unitymediagroup.de) Quit (Ping timeout: 480 seconds)
[22:02] * mykola (~Mikolaj@91.245.78.165) Quit (Quit: away)
[22:14] * derjohn_mobi (~aj@p578b6aa1.dip0.t-ipconnect.de) has joined #ceph
[22:15] * DV (~veillard@2001:41d0:a:f29f::1) Quit (Remote host closed the connection)
[22:16] * DV (~veillard@2001:41d0:a:f29f::1) has joined #ceph
[22:18] * vicente (~vicente@111-241-31-95.dynamic.hinet.net) has joined #ceph
[22:20] * dgurtner (~dgurtner@217.149.140.193) has joined #ceph
[22:20] * wwdillingham (~LobsterRo@140.247.242.44) has joined #ceph
[22:20] * georgem (~Adium@206.108.127.16) Quit (Quit: Leaving.)
[22:22] * _28_ria (~kvirc@opfr028.ru) has joined #ceph
[22:26] * vicente (~vicente@111-241-31-95.dynamic.hinet.net) Quit (Ping timeout: 480 seconds)
[22:26] <aarontc> xdr: you can build ceph on a lot smaller system than that.. just mind your -j flag so you don't run out of RAM
[22:27] <xdr> i ran it on a system with 4gb ram . It took forever to build .
[22:28] <aarontc> if build time is a concern, you can use binary packages. the Ceph repos have packages for ubuntu 14.04 and newer, and some other distros
[22:29] * Heliwr (~capitalth@hessel2.torservers.net) has joined #ceph
[22:29] <xdr> i needed to build it for hacking purposes . wanted to make some changes . built it on a 4gb machine last time , it took around 10 hours to build !
[22:29] * Hemanth_ (~hkumar_@103.228.221.142) Quit (Quit: Leaving)
[22:29] <xdr> do you think the above config should take lesser time ?
[22:30] <aarontc> I don't think the RAM was your limiting factor. I have a Pentium G4400 machine with 8GB of RAM that can build ceph in around 40 minutes
[22:30] * LDA (~lda@host217-114-156-249.pppoe.mark-itt.net) Quit (Quit: LDA)
[22:30] <aarontc> ceph relies heavily on advanced C++ features that make the compiler work very hard :)
[22:31] <xdr> ohh 1 okay ! thanks for the help . i guess i'll just have to build and see .
[22:33] <aarontc> Just make sure you use -j when you run make, with the number of CPU cores you have as the parameter. But don't go over (4GB RAM per job), so if you have 16GB of RAM, use -j4 even if you have 8 or 12 CPUs
[22:33] * linjan (~linjan@58PAACZZD.tor-irc.dnsbl.oftc.net) Quit (Remote host closed the connection)
[22:33] <aarontc> if you have a web browser open or something, might have to use less than 4 jobs even
[22:33] * linjan (~linjan@tor.yrk.urgs.uk0.bigv.io) has joined #ceph
[22:34] <xdr> This is what I have as CPU 2.5GHz dual-core Intel Core i5 processor (Turbo Boost up to 3.1GHz) with 3MB L3 cache . should this do ?
[22:34] * Hemanth (~hkumar_@103.228.221.142) has joined #ceph
[22:34] <aarontc> -j2 probably
[22:34] * Hemanth (~hkumar_@103.228.221.142) Quit ()
[22:35] * thomnico|Brussels (~thomnico@175.233-245-81.adsl-dyn.isp.belgacom.be) Quit (Ping timeout: 480 seconds)
[22:37] * angdraug (~angdraug@64.124.158.100) has joined #ceph
[22:37] <aarontc> I'd expect your build to take under 30 minutes if you're on a 3rd gen or newer i5 :)
[22:37] <xdr> :) . will let you know ! thanks for the help.
[22:38] <aarontc> no problem, good luck :)
[22:38] * thomnico (~thomnico@cable-46.253.163.149.coditel.net) has joined #ceph
[22:39] * diq (~diq@2620:11c:f:2:c23f:d5ff:fe62:112c) Quit (Quit: Leaving)
[22:39] * diq (~diq@2620:11c:f:2:c23f:d5ff:fe62:112c) has joined #ceph
[22:40] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[22:41] * csoukup (~csoukup@2605:a601:9c8:6b00:56f:e6a7:f22f:499f) has joined #ceph
[22:43] * DV (~veillard@2001:41d0:a:f29f::1) Quit (Remote host closed the connection)
[22:43] <Axion_Joey> Hi
[22:43] <Axion_Joey> in general should ceph's write speed increase by adding additional drives to existing hosts? or do I need to add more osd hosts to see an improvement?
[22:44] * DV (~veillard@2001:41d0:a:f29f::1) has joined #ceph
[22:45] * freakybanana (~freakyban@c-73-158-201-226.hsd1.ca.comcast.net) has joined #ceph
[22:46] <aarontc> Axion_Joey: depends on your CRUSH rules. if you have replicas that require different hosts you'll need more hosts to get more parallel write capacity
[22:46] <TMM> Axion_Joey, it depends on what is constrained right now, and the 'shape' of your cluster.
[22:46] <Axion_Joey> thanks for responding
[22:46] <TMM> Axion_Joey, basically what aarontc just said, for the 'shape'.
[22:46] <aarontc> :)
[22:46] <Axion_Joey> I have two osd hosts. and my rules write all data twice
[22:47] <TMM> Axion_Joey, regardless, if you are currently exceeding your network capacity you need to actually add more hosts (or upgrade the network hardware) likewise, if you are currently saturating your sata controller adding more disks won't help.
[22:47] <Axion_Joey> so in that case I won't see any performance improvements by adding more ssd's to the current hosts correct?
[22:47] <TMM> still depends on what is constrained now
[22:47] <Axion_Joey> write now I don't think I'm saturating either
[22:47] <Axion_Joey> I'm on a 10G network
[22:48] <Axion_Joey> per rados bench my mad bandwidth is 224MB'sec
[22:48] <Axion_Joey> sorry. meant max bandwidth
[22:48] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[22:48] * xdr (67334b61@107.161.19.53) Quit (Quit: http://www.kiwiirc.com/ - A hand crafted IRC client)
[22:48] <TMM> ok, you should still check out your cpu and sata/scsi links
[22:48] <TMM> if all of that is also under capacity adding a couple of ssds as a cache tier may be a good start
[22:49] <Axion_Joey> I'm using the LSI/Avago SAS 9211-i1
[22:49] <Axion_Joey> Thanks TMM
[22:49] <TMM> you should also really add more osd machines to your cluster. 2 is very little, if one of the hosts dies completely your data will be at serious risk
[22:49] <Axion_Joey> I wasn't sure if adding ssd's as a tier cache would help if the cluster is only made up of SSD's
[22:49] <TMM> oh, that depends then
[22:49] <TMM> if you have only a replicated pool it won't help
[22:49] <TMM> without any spinners
[22:50] <TMM> then just add them as regular osds
[22:50] <Axion_Joey> got it
[22:50] * skinnejo (~skinnejo@162.247.128.73) Quit (Ping timeout: 480 seconds)
[22:50] <TMM> but, again, if you have the funds to, I would highly recommend investing in a third host first
[22:50] <aarontc> Axion_Joey: just a word of caution if you're unaware, with two replicas you have no ceph-provided data integrity
[22:51] <TMM> I personally wouldn't run a cluster with less than 5 physical osd machines and 3 separate monitors
[22:51] <Axion_Joey> Does 224MB/s seem really low
[22:51] <Axion_Joey> ok wow
[22:51] <aarontc> Axion_Joey: 224MB/sec doesn't seem low, rados scales well with lots of horizontal resources
[22:51] <TMM> for 2 hosts? 224MB/s seems pretty alright
[22:51] <TMM> I get about 2.5GB/s on my 30 node cluster
[22:52] <Axion_Joey> got it. so it sounds like for small scale I would have been better off getting spinner drives and using the savings to add more hosts
[22:52] <TMM> for a single client btw, I can run multiple clients and get it multiples of 2.5GB/s
[22:52] <TMM> I haven't seen where it cuts off
[22:52] <aarontc> it's my experience that more hosts with fewer OSDs is better in general
[22:53] <Axion_Joey> Also regarding redundancy. I rebooted one osd, and the vm's running on the cluster seemed to work fine. When the machine came back up I saw ceph update the out of sync data. Does that mean that I still don't have any redundancy?
[22:53] <aarontc> Axion_Joey: you don't have integrity, you have redundancy
[22:53] <aarontc> if an OSD has bad data, there is no way to know *which* OSD of the two holding a given object
[22:53] <Axion_Joey> that makes total sense
[22:53] <TMM> and that will happen if you have enough disks
[22:53] <TMM> or enough data
[22:53] <aarontc> yep. as a rule I never run with n < 4 for replicated pools
[22:54] <Axion_Joey> our data needs are very low. We only need 1.2 TB's and our current writes on our production server max at 45M/s. We are evaluating ceph because of the real time failover
[22:55] <aarontc> I would still go with three hosts and replication factor 3...
[22:55] <aarontc> otherwise it will bite you sooner or later, IMHO
[22:55] <TMM> yes
[22:55] <Axion_Joey> ok. that's exactly what I was looking for. some advice from people that have actually done it. I truly appreciate it
[22:55] <aarontc> (unless you have backups that you trust and you verify integrity against regularly)
[22:55] <TMM> but you may be too late
[22:55] <aarontc> but it's probably safer to just grab a third host
[22:56] <TMM> the problem is that harddrives will just write garbage from time to time
[22:56] <TMM> they all do
[22:56] <aarontc> or read garbage :)
[22:56] <TMM> not very often, the specs for your drives will tell you how often you are expected to see that
[22:56] <TMM> but it is completely random
[22:56] <Axion_Joey> got it
[22:56] <aarontc> most drives are like 1 bit error in 10^20 or something.. however, given the size of current drives, and the number of IOs you perform, you *will* hit those errors on a regular basis (maybe days, months, weeks..)
[22:57] * shaunm (~shaunm@cpe-192-180-18-62.kya.res.rr.com) Quit (Ping timeout: 480 seconds)
[22:57] <aarontc> for my cluster of 16 OSD hosts with 48 OSDs, I hit bad reads from spinning disks at least once a month
[22:57] <TMM> we have 300 1TB drives with a MTBF of 1.5M hours
[22:57] <TMM> how often does a drive fail?
[22:58] <Axion_Joey> wow that often?
[22:58] <TMM> roughly once ever 180 days a drive just dies for no reason
[22:58] <TMM> these odds all stack :)
[22:58] <aarontc> TMM: you should have one drive fail at least once per 6.7 months
[22:58] <TMM> aarontc, pretty much :)
[22:58] <aarontc> yay statistics :)
[22:58] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit (Ping timeout: 480 seconds)
[22:58] * georgem (~Adium@24.114.66.245) has joined #ceph
[22:58] <T1> I've seen disks last 6+ years running 24/7, and I've seen then fail any time before that..
[22:58] <Axion_Joey> our storage and io needs have always been really low. we've always sata drives on the vm hosts. and the performance has been fine. but the servers are getting old so we thought that it was a perfect time to move to shared storage
[22:59] * Heliwr (~capitalth@4MJAAD5V0.tor-irc.dnsbl.oftc.net) Quit ()
[22:59] <TMM> aarontc, I was trying to explain to Axion_Joey that 10^20 sounds unlikely but that this stuff stacks
[22:59] * georgem (~Adium@24.114.66.245) Quit (Read error: Connection reset by peer)
[22:59] <Axion_Joey> I get that
[22:59] <TMM> most people see 1 in 1.5 million or 1 in 10^20 and think that it'll 'never happen'
[22:59] <aarontc> Axion_Joey: shared storage is pretty awesome - it just takes a bit more than people tend to realize (with ceph at least)
[22:59] * georgem (~Adium@24.114.79.162) has joined #ceph
[22:59] <Axion_Joey> we also decided to use ssd's just because they're (depending on who you ask) more reliable than spinner disks
[22:59] * georgem (~Adium@24.114.79.162) Quit (Read error: Connection reset by peer)
[23:00] <aarontc> there are other storage systems which do more overhead internally to give you integrity with only two copies, but none are FOSS that I know of
[23:00] <Axion_Joey> but we don't need the performance really
[23:00] <Axion_Joey> I might be better off trying to get amazon to let me return the drives I've already bought
[23:00] <TMM> aarontc, I guess if you have your OSDs on btrfs you get at least some security with just 2 hosts
[23:00] <Axion_Joey> right now they're on xfs
[23:01] <Axion_Joey> but I can test btrfs. but from what I've read no one recommends it for production yet
[23:01] <T1> ssd more reliable than ssd? eeeks..
[23:01] <TMM> Axion_Joey, btrfs will at least do checksumming of expected data, so you can still scrub with just 2 osds
[23:01] <T1> ehm..
[23:01] <T1> than sas/sata .. eeek
[23:01] <T1> it really really REALLY depends on the ssd
[23:02] <TMM> Axion_Joey, I've been running btrfs in production for at least a year now. Just make sure you get a kernel > 3.16 and don't try anything fancy with raid5 until > 4.2
[23:02] * wwdillingham (~LobsterRo@140.247.242.44) Quit (Ping timeout: 480 seconds)
[23:02] <Axion_Joey> TMM that's good to know
[23:02] * evelu (~erwan@37.165.41.67) Quit (Ping timeout: 480 seconds)
[23:02] <T1> given the horror stories I've seen with failing ssds in ceph I'd be very careful with using just any plain consumer ssd
[23:03] <Axion_Joey> T1 I bought the SanDISK extreme pro's
[23:03] <TMM> you really want an SSD with a capacitor that can write out internal accounting state on power failure
[23:03] <TMM> those are only about 15% more expensive than consumer drives
[23:03] <TMM> that's really all you need
[23:03] <TMM> I run about 300 of these: http://www.samsung.com/semiconductor/products/flash-storage/enterprise-ssd/MZ7LM960HCHP
[23:03] <T1> Axion_Joey: that sounds wrong..
[23:03] <TMM> it hasn't been a problem
[23:04] <Axion_Joey> a friend of mine just bought a $200K Tegile SAN and they used the same drives
[23:04] * icey (~Chris@0001bbad.user.oftc.net) Quit (Remote host closed the connection)
[23:05] <T1> PM863 is new.. if and when you've had them running for 3+ years with no incident I'd say they are ok - for now samsung is too unproven for my taste
[23:05] <m0zes> we just went through a rash of failures. 1 a day for a week and a controller failure. out of our 432 osds.
[23:05] <Axion_Joey> wow
[23:05] <Axion_Joey> m0zes what drives were you using?
[23:05] <T1> TMM: but only 1.3 DWPD?
[23:05] <TMM> T1, so far they have been failing at the expected rate :)
[23:06] <m0zes> seagate 6TB NL SAS disks in our dell machines.
[23:06] <m0zes> the machines are about a year old.
[23:06] <T1> m0zes: uff..
[23:06] <TMM> T1, 0.3 for 4K writes. We're estimating about 0.5 DWPD for a RBD workload
[23:06] <darkfader> the summed up version is you need to treat any ssd as a wearing part
[23:06] <Axion_Joey> makes sense
[23:06] <TMM> T1, on our cluster that is about 10TB of writes a day for the warranty period of the drives.
[23:07] <T1> TMM: hm, ok - but I'd still like to hear what you mean in a few years.. :)
[23:07] * haplo37 (~haplo37@199.91.185.156) Quit (Read error: Connection reset by peer)
[23:07] <T1> or.. what you think of those drives
[23:07] <TMM> At the very low end, assuming some OSDs will be getting hit with writes 4x harder than others
[23:07] <Axion_Joey> one more newbie question (for now at least). If we go from 2 osd server and write data twice, to 3 osd servers and write data 3 times, will we see an increase in write performance, or just gain the advantage of data integrity?
[23:07] <TMM> which is double the spread that we actually see
[23:07] <T1> (sorry - I'm getting tired and my fingers are not hitting the right keys..)
[23:08] <TMM> :)
[23:08] <T1> TMM: well.. at least they're DC-level drives
[23:08] <T1> :)
[23:08] <TMM> T1, yeah, the difference being really only that capacitor
[23:08] <T1> oh yes
[23:09] <darkfader> yeah at least you won't have to explain being cheap after a large power blackout
[23:09] <T1> one of the reasons I also went with intel S3710s
[23:09] <Axion_Joey> T1 I think your budget is waaaaaaay bigger than mine
[23:09] <T1> Axion_Joey: they were .. 500DKK more expensive than what we originally expected..
[23:09] <T1> 500DKK is.. a little below 100USD
[23:10] <TMM> darkfader, I'm not too worried about that. The caps should keep the SSDs themselves consistent, xfs should keep the osd filesystems consistent and the osd journals should keep the osd data consistent.
[23:10] <darkfader> TMM: that's what i mean
[23:10] <TMM> darkfader, I'm really not worried about a power outage causing more problems then you'd expect from a power outage
[23:10] <Axion_Joey> T1 Wow.
[23:10] <Axion_Joey> I'm going to have a hard enough time just buying a third osd host
[23:10] <darkfader> TMM: people going with consumer drives would be in a less fun situation there
[23:10] <T1> we originally thought of samsung 850 pro, but.. no..
[23:10] <T1> nooooo no
[23:10] <Axion_Joey> wow
[23:10] <TMM> darkfader, yes, they would be what we call 'fucked'
[23:11] <darkfader> and responsible
[23:11] <TMM> especially with a lot of drives
[23:11] <janos_> but it says "Pro" on it!! :p
[23:11] <T1> yes ceph CAN run on comodity hardware, but..
[23:11] <T1> 10gbit network and proper ssds if you want anything good from it
[23:11] <TMM> if Axion_Joey has a decent UPS setup with graceful shutdowns it's not an issue though
[23:12] <TMM> that is, at least a separate UPS per failure domain
[23:12] <TMM> in Axion_Joey's case that would be one per physical osd host
[23:12] <TMM> you definitely CAN make a ceph cluster out of really cheap materials and not have issues
[23:12] <T1> of course
[23:13] <TMM> but the cheaper you go the more holistic your approach has to be
[23:13] <T1> buuut
[23:13] * skinnejo (~skinnejo@173-27-199-104.client.mchsi.com) has joined #ceph
[23:13] <T1> I'm not going to have any of that on my watch
[23:13] <Axion_Joey> I'm not really too worried about data integrity. On the front end we write data indepently to 2 different data centers. But our current storage is slow, but consistently measures 240MB/s. The fact that the SAN with SSD drives is only performing at 200MB/s is causing me all kinds of problems
[23:13] * skinnejo (~skinnejo@173-27-199-104.client.mchsi.com) Quit ()
[23:13] <TMM> T1, sure, it also depends on your size. If your cluster is small enough that you can actually make full *backups* of it, you can slack even further.
[23:14] <TMM> T1, I'm not going to be backing up 200tb of data myself.
[23:14] <T1> haha
[23:14] <T1> we actually have a full backup of everything in ceph.. ;)
[23:14] <TMM> For my usecase that's up to the users
[23:15] <T1> but ok - we have a pretty small cluster, so it's easily done
[23:15] <T1> we just do not want to loose anything..
[23:15] * wjw-freebsd (~wjw@smtp.digiware.nl) has joined #ceph
[23:15] <Axion_Joey> We only need 1.2TB. I can backup on a USB drive
[23:16] <TMM> T1, the new cluster that's being installed will be 1PB big :P
[23:16] <Axion_Joey> I just won't be able to get buyoff to deploy ceph if I can't make it go at least as fast as oru current storage
[23:16] <TMM> T1, DEFINITELY not backing that up
[23:16] <T1> our customers data are on it - and the jobs for some 100 coworkers
[23:16] <T1> TMM: mmmmmmmmm
[23:17] <darkfader> Axion_Joey: you might wanna get a leased set of servers for a month then
[23:17] <darkfader> or at least borrow two boxes with ssd's from a friendly vendor
[23:17] <TMM> T1, we run almost entirely rbd on the cluster through cinder/nova. We have cinder-backup to some other devices that users can pay for. We explicitly make no guarantees about data in nova root disks
[23:17] <darkfader> how else you wanna prove that up front.
[23:17] <T1> TMM: just emagine the amount of storage needed for 1PB of backups.. the sheer number of shelves
[23:17] <TMM> T1, yeah, not doing it :P
[23:18] <T1> imagine even
[23:18] <T1> haha
[23:18] <T1> TMM: well.. our users don't have a clue how or where data resides
[23:18] * georgem (~Adium@24.114.72.94) has joined #ceph
[23:18] <Axion_Joey> darkfader I have a third server I can use here for testing
[23:18] <T1> just as long as it's available
[23:18] <Axion_Joey> I was just wondering if we should see a performance increase if we have 3 servers, but still write all data 3 times.
[23:18] <TMM> T1, we have 10 of these : http://www.supermicro.nl/products/nfo/Simply_Double.cfm
[23:18] <TMM> T1, for the 1pb cluster
[23:18] * vicente (~vicente@111-241-31-95.dynamic.hinet.net) has joined #ceph
[23:19] <janos_> woah sweet
[23:19] <TMM> T1, about 1/3rd SSDs, 2/3rds spinners
[23:19] <TMM> 4x10GB connectivity
[23:19] <T1> nice
[23:19] <T1> I thought of doing something with a bunch of Dell MD1280s
[23:20] <T1> http://www.dell.com/us/business/p/storage-md1280/pd
[23:20] <TMM> SM has some really high-density boxes too
[23:20] <T1> 84x 8TB drives
[23:20] <T1> in 4U
[23:20] <T1> oh, 5U - sorry
[23:20] <TMM> http://www.supermicro.nl/products/nfo/storage.cfm
[23:21] <T1> but yes.. I'd still require a few servers for that
[23:21] <TMM> 90-bay in 4u
[23:21] <T1> mmm
[23:21] <TMM> I was considering them, but decided that 2x48bay in 4u was better for redundancy
[23:21] <T1> the only problem with that is the rather large failure domain
[23:21] <TMM> and I'd get twice the cpus
[23:22] <TMM> yeah, that's why I ultimately decided against them
[23:22] <T1> buuut then again..
[23:22] <T1> there's room for expansion
[23:22] <TMM> you'd need at least 5 of the fuckers
[23:22] <T1> yeah.. :D
[23:22] <T1> just start out with 30 drives in each..
[23:22] <TMM> and then you have 90 OSDs per host
[23:23] <TMM> I though that 48 was already pushing it a bit
[23:23] <T1> heh, yeah
[23:23] <T1> juuust a tad
[23:23] <TMM> that's already 72000 threads per node
[23:23] <T1> 2 or 4GB ram per TB per OSD?
[23:23] <TMM> didn't want to try for 1.5 million :P
[23:23] <T1> haha
[23:24] <T1> no
[23:24] <TMM> we have 256gb of ram per osd
[23:24] <T1> and if one goes down..
[23:24] <T1> wow - replacement for 90 OSDs at once
[23:24] <TMM> about 5gb per OSD
[23:24] <T1> on all other nodes..
[23:25] <TMM> so yeah, somewhere between 2 and 4gb per tb per osd
[23:25] <T1> ok
[23:25] <T1> seems resonable
[23:25] * olid19811117 (~olid1982@ip503c3d0c.speed.planet.nl) Quit (Ping timeout: 480 seconds)
[23:27] * vicente (~vicente@111-241-31-95.dynamic.hinet.net) Quit (Ping timeout: 480 seconds)
[23:27] <TMM> it should be fine I think
[23:27] * icey (~Chris@pool-74-109-7-163.phlapa.fios.verizon.net) has joined #ceph
[23:27] <TMM> even our ec pools don't seem to chew through too much ram
[23:27] <TMM> even during backfill
[23:27] <TMM> it seems that the 1GB per tb per osd is already on the cautious side
[23:28] <aarontc> I've had a hard time getting any OSD process to use more than 2GB of RAM total, regarldess of storage size
[23:28] <aarontc> (I have from 100GB to 5TB OSDs)
[23:28] <T1> the mailinglists says otherwise..
[23:28] <TMM> Maybe it depends on your ec algo
[23:28] <T1> the recommendation has risen up from 1 to 2
[23:28] <T1> just to be safe..
[23:29] <TMM> sure, and I bought a safetely margin on top of that even
[23:29] <TMM> I'm not suicidal
[23:29] <TMM> it just seems that you can probably get away with less if you're really strapped for cash
[23:29] <aarontc> I still allocated 1GB/TB of storage on each node, I just see it sitting idle, even during recoveries. the OS-level block caching is not going to waste, I'm sure
[23:29] <TMM> and are willing to take a risk on extreme slowness during extreme backfill scenarios
[23:30] * vbellur (~vijay@71.234.224.255) has joined #ceph
[23:33] * georgem (~Adium@24.114.72.94) Quit (Quit: Leaving.)
[23:33] * maku (~Unforgive@tor-exit4-readme.dfri.se) has joined #ceph
[23:37] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[23:37] * bene2 (~bene@nat-pool-bos-t.redhat.com) Quit (Quit: Konversation terminated!)
[23:41] * brad_mssw (~brad@66.129.88.50) Quit (Quit: Leaving)
[23:43] * icey (~Chris@0001bbad.user.oftc.net) Quit (Remote host closed the connection)
[23:46] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[23:48] * icey (~Chris@pool-74-109-7-163.phlapa.fios.verizon.net) has joined #ceph
[23:55] * dkrdkr (48a3dc03@107.161.19.53) Quit (Quit: http://www.kiwiirc.com/ - A hand crafted IRC client)
[23:56] * thomnico (~thomnico@cable-46.253.163.149.coditel.net) Quit (Read error: Connection reset by peer)
[23:59] * EinstCrazy (~EinstCraz@116.225.239.138) Quit (Remote host closed the connection)

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.