Icehouse Design Summit

This session will include the following subject(s):

Big Data processing:

Large deployments may produce huge volume of stats. To see it graphically you may play with http://docs.openstack.org/developer/ceilometer/install/dbreco.html: if there are 20000 instances and probe happens once a minute we will get 1TB per month. Besides, some of metering is better to be probed more frequently and it will increase amount of stored data. At such scale we need to use BigData technologies and Apache Hadoop is one of them. It can work with all backends from Ceilometer: with SQL and DB2 using Apache Sqoop, with Mongo using Hadoop-Mongo connector and HBase. At this session I will share my team's experience in solving metering problems using HBase and Hadoop.

(Session proposed by Nadya Privalova)

Scaling Ceilometer:

Certain meters need to be able to check and provided data on the health and status of workloads, storage, network, etc. nearer to real-time in order for automated (or manual) notifications to be generated against policies to fix "problems" as they occur. Currently, Ceilometer defaults to collect meters at 10minute intervals but it is foreseeable that there are meters that need to be captured at a much finer granularity (in the milliseconds). Additionally, Ceilometer continues to support/record more and more data.

This design session is to discuss how Ceilometer scales to handle increasing amount of data captured by increasing amount of items it meters/monitors/alarms. How do we design/implement Ceilometer to handle various stress levels on both the collector and backend (and what are the limits Ceilometer is expected to handle).

Ceilometer has the ability to deploy multiple collectors horizontally... is that enough and is there a better way? Also, what are the limits to what Ceilometer should collect (ie. collecting logging information will increase data load dramatically).

(Session proposed by gordon chung)