879822 - Store Bugzilla data in Elastic Search

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Mark Côté [:mcote]]

This is a tracking bug for the steps required to set up an ElasticSearch database containing the same kind of data as metrics' internal cluster, but publicly available and thus restricted to public bugs only.

Background copied from bug 872363:

metrics has an ES cluster that contains not only the current state of Bugzilla, but past history as well.  ElasticSearch provides a better method of performing certain types of analyses, and use of it doesn't impact the main Bugzilla database.  However, it contains *some* information on confidential bugs, so it is behind LDAP, which means that any tools that use it generally must be behind LDAP as well, which limits their usefulness.

It would thus be really great to have a publicly accessible ES cluster which contains *only* data on public bugs (or to word it another way, *no* data on confidential bugs of any type).

mcoates has already signed off on this idea from a security standpoint, provided that bugs with any limitations on visibility are excluded, and that all data concerning bugs that are made confidential some time after creation (i.e. were at one point public, but no longer) is removed in a timely fashion.

Comment 1

[Mark Côté [:mcote]]

Redefining this bug a little to reflect expanded scope.

Since metrics would like to decommission their Bugzilla ES cluster entirely, we will need a replacement.  This will be a FULL mirror of Bugzilla data, including confidential bugs, and hence any applications using it MUST be behind LDAP.  This is how the current metrics-run cluster works.

This is in addition to the cluster containing only public bugs.  We can use the same ETL software for both; I'm not sure if we will need two separate machines for adequate processing power, though.  Filing request for machine separately.

Comment 2

[Josh Matthews [:jdm]]

What will it mean to be behind LDAP? What LDAP accounts will have access to it?

Comment 3

[Kyle Lahnakoski [:ekyle]]

Josh,

There are now two parts.  The first is public with confidential bugs removed on ES cluster.   

The second ES cluster includes confidential bugs, but with summary and comments removed.   This second one will remain behind LDAP.  I believe anyone with LDAP access should be allowed access to this data, just like the current Metrics setup.  Security may have another opinion.

Both clusters will be fed by one common piece of ETL code (https://github.com/klahnakoski/Bugzilla-ETL)

Comment 4

[Mark Côté [:mcote]]

Yeah, as Kyle said, right now the metrics cluster is accessible by anyone with an LDAP account.  We can revisit this if necessary.  At least Kyle and I will need access to it for starters.

Comment 5

[Kyle Lahnakoski [:ekyle]]

Attachment: Architecture Drawing (v1)

Please add suggestions

Comment 6

[Mark Côté [:mcote]]

Comment on attachment 828667 [details]
Architecture Drawing (v1)

I think this is good--however I would much prefer the diagram to be in a standard picture format like PNG.

Comment 7

[Mark Côté [:mcote]]

Just found out that we *can* create Gliffy diagrams in mana; you just have to have the correct permissions for the space.  I suggest creating your own space (in the drop-down, top right) and using Gliffy to publish the diagram there.

Comment 8

[Kyle Lahnakoski [:ekyle]]

Attachment: Architecture Drawing (v2)

Added port numbers, and client/server relationships.  Showed LDAP machines as cloud, and Kyle's ES Cluster specifically.

Comment 9

[Mark Côté [:mcote]]

This looks a little off in LibreOffice.  Can you export to a PNG?

Comment 10

[Kyle Lahnakoski [:ekyle]]

Attachment: Architecture v3

png is easier to view

Comment 11

[Kyle Lahnakoski [:ekyle]]

\o/

An example of a public dashboard:  http://claw.cs.uwaterloo.ca/~okononen/